Buckets:

hf-doc-build/doc-dev / diffusers /pr_15244 /en /api /models /ltx2_diffusion_decoder.md
|
download
raw
17.9 kB

LTX2VideoDiffusionDecoderModel

The diffusion video decoder introduced in LTX-2.5 by Lightricks. Neighborhood-attention stages upsample the latent into a context volume, and a final stage denoises pixels conditioned on that context.

It is a decoder, not an autoencoder: encoding stays with AutoencoderKLLTX2Video, whose latent space this consumes unchanged, so latents are interchangeable between the convolutional decoder and this one. Because it is itself a diffusion model it is driven by LTX2VideoDiffusionDecodePipeline rather than being passed as a pipeline's vae: run any LTX-2 pipeline with output_type="latent", then decode.

import torch
from diffusers import LTX2Pipeline, LTX2VideoDiffusionDecodePipeline, LTX2VideoDiffusionDecoderModel

pipe = LTX2Pipeline.from_pretrained("Lightricks/LTX-2.5-Diffusers", dtype=torch.bfloat16).to("cuda")  # or "mps", "xpu", "cpu"
latents = pipe(prompt="a potter shaping a clay vase", output_type="latent").frames

decoder = LTX2VideoDiffusionDecoderModel.from_pretrained(
    "Lightricks/LTX-2.5-Diffusers", subfolder="diffusion_decoder", dtype=torch.bfloat16
).to("cuda")
decode_pipe = LTX2VideoDiffusionDecodePipeline(diffusion_decoder=decoder, scheduler=pipe.scheduler)

# `denormalize=False`: `output_type="latent"` already applied the latent statistics, so applying them
# again here would scale every channel by its std a second time.
# The decoder also draws the noise it denoises, so decoding is only reproducible with a generator.
video = decode_pipe(
    latents, generator=torch.Generator("cuda").manual_seed(0), denormalize=False
).frames[0]

vae is an optional component on the decode pipeline: it is only consulted for the latent statistics when denormalize=True, and the decoder carries its own, so a decode-only workflow does not have to load a second autoencoder.

Attention backends

The neighborhood-attention window is expressed as a BlockMask, so the decoder runs on the flex attention backend by default and needs no extra dependency. PyTorch does not compile flex_attention unless you ask it to, and uncompiled it materializes the full score matrix — which is impractical at full-resolution sequence lengths. For those, either compile the decoder or switch to NATTEN's kernels, which are also what the original implementation uses. The processor fetches NATTEN from the Hub (shi-labs/natten) through the kernels package, so it needs pip install kernels rather than a local NATTEN build:

from diffusers.models.autoencoders.ltx2_diffusion_decoder import LTX2VideoVaeNeighborhoodNattenProcessor

decoder.set_attn_processor(LTX2VideoVaeNeighborhoodNattenProcessor())

Fetching the kernel downloads code from the Hub, so the processor raises when remote code is disabled globally with DIFFUSERS_DISABLE_REMOTE_CODE=true.

Every attention module in the decoder is the same neighborhood attention (per-stage differences like the kernel size live on the module, not the processor), so set_attn_processor swaps them all with one shared instance.

Switching the backend (decoder.set_attention_backend(...)) to anything but flex raises: no other backend accepts the BlockMask. Use the NATTEN processor above instead.

Tiling

decoder.enable_tiling() decodes in overlapping tiles that are blended back together, bounding peak memory by the tile size instead of the video size. The cheap early upsampling stages still see the full latent — only the last upsampling stage and the diffusion stage, which dominate decode memory, run per tile — so tiling changes the output only near tile borders. Because the diffusion stage denoises each tile separately, a tiled decode does not reproduce the untiled result exactly; the default tile and overlap sizes match the reference implementation's. Neighborhood attention rejects any grid smaller than its kernel, so a trailing remnant tile is merged into its neighbor rather than decoded on its own.

Keyframe-aware decoding

A decode can be anchored on keyframe planes: single-frame latents at known pixel frames, for example the generated slots of LTX2DFRPipeline. Each plane must be the latent of a standalone one-frame clip (the VAE is causal, so a frame encoded inside a clip is a different latent), denormalized like the video latents. Positions are pixel-frame indices, not latent-frame indices.

Omit both arguments for a plain decode; that path does not depend on whether the checkpoint carries a keyframe tag.

video = decoder.decode(
    latents,  # (B, C, F, H, W), already denormalized
    generator=torch.Generator("cuda").manual_seed(0),
    keyframe_latents=keyframe_latents,  # (B, C, P, H, W)
    keyframe_frame_indices=[24, 48],  # pixel frame of each plane
).sample

The planes go through the same weights as the video. In every attention layer each video position also attends to the same spatial window on its two nearest planes (frames before the first plane attend to that plane alone), and each plane to the same window on its two nearest frames. This joint attention runs on a built-in PyTorch implementation whatever attention processor is set, since neither FlexAttention's block mask nor NATTEN expresses it.

Checkpoints trained for keyframe decoding also carry a learned tag added to the plane latents, which decoder_keyframe_type_embedding=True creates as decoder.type_emb. Decoding with keyframe planes on a checkpoint that lacks that tag raises rather than decoding the planes untagged. If a checkpoint's weights carry decoder.type_emb but its config does not set the flag, pass decoder_keyframe_type_embedding=True to LTX2VideoDiffusionDecoderModel.from_pretrained(); otherwise the tag is dropped as an unexpected key. Converting the original VAE with convert_ltx2_diffusion_video_vae sets the flag when the tag is present.

With tiling enabled each temporal tile keeps the planes inside it plus the nearest plane on each side.

Standard pipeline

LTX2VideoDiffusionDecodePipeline forwards the same arguments. A DFR pass with output_type="latent" already denormalized both the video and the slots:

from diffusers import LTX2DFRPipeline, LTX2VideoDiffusionDecodePipeline, LTX2VideoDiffusionDecoderModel

dfr = LTX2DFRPipeline.from_pretrained("Lightricks/LTX-2.5-Diffusers", dtype=torch.bfloat16)
out = dfr(prompt="...", output_type="latent", generate_slots=True)

decoder = LTX2VideoDiffusionDecoderModel.from_pretrained(
    "Lightricks/LTX-2.5-Diffusers",
    subfolder="diffusion_decoder",
    dtype=torch.bfloat16,
    decoder_keyframe_type_embedding=True,  # keep the trained keyframe tag `decoder.type_emb`
)
decode_pipe = LTX2VideoDiffusionDecodePipeline(diffusion_decoder=decoder, scheduler=dfr.scheduler)
video = decode_pipe(
    out.frames,
    generator=torch.Generator("cuda").manual_seed(0),
    denormalize=False,
    keyframe_latents=out.keyframes,
    keyframe_frame_indices=out.keyframe_positions,
).frames

Modular workflow

LTX2DiffusionVaeDecoderStep (the LTX-2.5 decode block) declares the same inputs. Packed denoised latents are unpacked and denormalized as usual; keyframe_latents are already denormalized 5-D planes, matching LTX2DFRPipelineOutput.keyframes. They are optional, so they do not change automatic workflow selection.

from diffusers.modular_pipelines.ltx2.decoders import LTX2DiffusionVaeDecoderStep

decode = LTX2DiffusionVaeDecoderStep().init_pipeline("Lightricks/LTX-2.5-Diffusers")
# Keep the trained keyframe tag `decoder.type_emb`; the dict form passes the flag to the decoder only.
decode.load_components(dtype=torch.bfloat16, decoder_keyframe_type_embedding={"diffusion_decoder": True})
videos = decode(
    latents=packed_latents,
    height=512,
    width=704,
    num_frames=121,
    dtype=torch.bfloat16,
    generator=torch.Generator("cuda").manual_seed(0),
    keyframe_latents=out.keyframes,
    keyframe_frame_indices=out.keyframe_positions,
    output="videos",
)

LTX2VideoDiffusionDecoderModel[[diffusers.LTX2VideoDiffusionDecoderModel]]

diffusers.LTX2VideoDiffusionDecoderModel[[diffusers.LTX2VideoDiffusionDecoderModel]]

diffusers.LTX2VideoDiffusionDecoderModel(out_channels: int = 3, latent_channels: int = 128, patch_size: int = 4, scaling_factor: float = 1.0, decoder_head_dim: int = 64, decoder_stage_channels: tuple[int, ...] = (2048, 1024, 512, 512, 256), decoder_stage_depths: tuple[int, ...] = (4, 6, 4, 2, 8), decoder_stage_kernels: tuple[tuple[int, int, int], ...] = ((3, 7, 7), (3, 7, 7), (3, 5, 5), (3, 5, 5)), decoder_upsample_strides: tuple[tuple[int, int, int], ...] = ((1, 2, 2), (2, 1, 1), (2, 2, 2), (2, 2, 2)), decoder_upsample_channel_reductions: tuple[int, ...] = (2, 2, 1, 2), decoder_stage5_kernel: tuple[int, int, int] = (11, 11, 11), decoder_t_emb_dim: int = 384, decoder_timestep_scale_multiplier: float = 1000.0, decoder_model_output_type: str = 'x0', decoder_num_inference_steps: int = 1, spatial_compression_ratio: int = 32, temporal_compression_ratio: int = 8, decoder_keyframe_type_embedding: bool = False)

Source

The LTX-2 diffusion video decoder, introduced in LTX-2.5.

This is a decoder, not an autoencoder: it has no encoder and cannot produce latents. Encoding stays with AutoencoderKLLTX2Video, whose latent space this consumes unchanged, so latents are interchangeable between the convolutional decoder and this one.

It is also a diffusion model rather than a deterministic decoder — it denoises pixels conditioned on a context volume built from the latents — which is why it is driven by LTX2VideoDiffusionDecodePipeline rather than being passed as a pipeline's vae.

The latent statistics are carried here as buffers so the decode pipeline can denormalize without loading a second autoencoder just for two vectors.

Decoding can be anchored on keyframe planes: single-frame latents at known pixel frames, each encoded as a standalone one-frame clip, passed as keyframe_latents / keyframe_frame_indices to decode. Every video position then also attends to the same spatial window on its two nearest planes, or on the first plane alone for frames before it. Checkpoints trained for it carry a learned tag added to the plane latents, decoder.type_emb, which decoder_keyframe_type_embedding=True creates. Decoding with keyframe planes on a checkpoint that lacks that tag raises rather than decoding them untagged.

This model inherits from ModelMixin. Check the superclass documentation for it's generic methods implemented for all models (such as downloading or saving).

decode[[diffusers.LTX2VideoDiffusionDecoderModel.decode]]

decode(z: Tensor, generator: typing.Optional[torch.Generator] = None, num_inference_steps: int | None = None, return_dict: bool = True, keyframe_latents: typing.Optional[torch.Tensor] = None, keyframe_frame_indices: typing.Optional[torch.Tensor] = None)

Source

Decode a batch of latents.

z is expected to be denormalized already (the pipeline applies latents_mean / latents_std), matching AutoencoderKLLTX2Video. This decoder denoises, so pass generator for reproducibility.

keyframe_latents (B, C, P, H, W), denormalized like z, anchors the decode on P keyframe planes at the pixel frames keyframe_frame_indices (P,). Each plane must be the latent of a standalone one-frame clip. Planes may lie outside the decoded clip: they are ranked by temporal distance. The checkpoint must carry decoder.type_emb (decoder_keyframe_type_embedding=True); a decode with planes otherwise raises.

enable_tiling[[diffusers.LTX2VideoDiffusionDecoderModel.enable_tiling]]

enable_tiling(tile_sample_min_height: int | None = None, tile_sample_min_width: int | None = None, tile_sample_min_num_frames: int | None = None, tile_sample_stride_height: int | None = None, tile_sample_stride_width: int | None = None, tile_sample_stride_num_frames: int | None = None)

Source

Parameters:

tile_sample_min_height (int, optional) : The height of one decoded tile, in pixels.

tile_sample_min_width (int, optional) : The width of one decoded tile, in pixels.

tile_sample_min_num_frames (int, optional) : The number of frames of one decoded tile.

tile_sample_stride_height (int, optional) : The distance in pixels between the tops of two consecutive vertical tiles; the difference to tile_sample_min_height is the blended overlap.

tile_sample_stride_width (int, optional) : The distance in pixels between the left edges of two consecutive horizontal tiles.

tile_sample_stride_num_frames (int, optional) : The distance in frames between the starts of two consecutive temporal tiles.

Enable tiled decoding. The deterministic upsampling stages before the last one always process the full latent (they run at low resolution and are cheap); the last stage and the stage-5 diffusion blocks — which dominate decode memory — run on overlapping tiles whose seams are blended linearly.

disable_tiling[[diffusers.LTX2VideoDiffusionDecoderModel.disable_tiling]]

disable_tiling()

Source

Disable tiled decoding, returning to decoding the whole video in one pass.

forward[[diffusers.LTX2VideoDiffusionDecoderModel.forward]]

forward(z: Tensor, generator: typing.Optional[torch.Generator] = None, num_inference_steps: int | None = None, return_dict: bool = True, keyframe_latents: typing.Optional[torch.Tensor] = None, keyframe_frame_indices: typing.Optional[torch.Tensor] = None)

Source

Parameters:

z (torch.Tensor) : Latents of shape (B, C, F, H, W), expected to be denormalized already (the pipeline applies latents_mean / latents_std), matching AutoencoderKLLTX2Video.

generator (torch.Generator, optional) : This decoder denoises, so pass a generator to make decoding reproducible.

num_inference_steps (int, optional) : Number of denoising steps. Defaults to the decoder's decoder_num_inference_steps config value.

return_dict (bool, optional, defaults to True) : Whether to return a DecoderOutput instead of a plain tuple.

keyframe_latents (torch.Tensor, optional) : Keyframe planes of shape (B, C, P, H, W), denormalized like z and on its latent grid, one latent frame per plane, each encoded as a standalone one-frame clip. Every video position also attends to the same spatial window on its two nearest planes, or on the first plane alone for frames before it.

keyframe_frame_indices (torch.Tensor, optional) : The (P,) pixel frame of each plane in the decoded video. Required with keyframe_latents.

Returns: DecoderOutput or tuple

tiled_decode[[diffusers.LTX2VideoDiffusionDecoderModel.tiled_decode]]

tiled_decode(z: Tensor, generator: typing.Optional[torch.Generator] = None, num_inference_steps: int | None = None, keyframe_latents: typing.Optional[torch.Tensor] = None, keyframe_frame_indices: typing.Optional[torch.Tensor] = None)

Source

Decode a batch of latents with the last deterministic stage and the diffusion stage running per tile.

Tiles live on the grid entering the last deterministic stage, where one cell maps to a fixed block of output pixels; the tile_sample_* sizes are converted to that grid, so they should be multiples of the cell size (8 px spatially and 2 frames temporally for the production config). Temporal tiles follow the causal frame mapping: the tile containing t=0 drops the temporal upsample's duplicate leading frame and only the tile containing the video end carries the NATTEN border padding.

With keyframe planes, each temporal tile carries the planes inside its pixel-frame span plus the nearest one on each side of it, with their times rebased on the tile's first cell and first pixel frame, and each spatial tile crops the planes with the same slices as the video.

Xet Storage Details

Size:
17.9 kB
·
Xet hash:
40fbb940325e29439c0bcc01213badd3c8eb283085d22277b333ac06583b8fcf

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.