Buckets:
SANA-WM
SANA-WM is a camera-controlled image-to-video world model built on top of SANA. Given a first-frame image, a text
prompt, and a camera trajectory (either explicit c2w poses or a WASD/IJKL action string), it generates a video
whose motion follows the requested camera path.
Inference runs in two stages:
- Stage 1 — SANA-WM DiT. A 1.6B-parameter bidirectional DiT with GDN-Triton linear attention and a UCPE camera-control branch. Sampling uses an LTX-style flow-matching Euler scheduler with per-token timesteps; the first latent frame is the conditioning anchor.
- Stage 2 — LTX-2 refiner (optional). A separate sink-bidirectional Euler refiner pipeline
(SanaWMLTX2Refiner) that wraps
SanaWMLTX2RefinerTransformer3DModel +
LTX2TextConnectorsand a Gemma-3 text encoder, run for 3 distilled sigma steps.
Both stages decode through the AutoencoderKLLTX2Video VAE.
Available models:
| Model | Recommended dtype |
|---|---|
Efficient-Large-Model/SANA-WM_bidirectional-diffusers |
torch.bfloat16 |
Efficient-Large-Model/SANA-WM_bidirectional-diffusers-refiner |
torch.bfloat16 |
SANA-WM is trained at a fixed 704×1280 resolution. The recommended dtype is for the transformer weights — keep the text encoder in
torch.bfloat16and the VAE intorch.float32for best numerics. The pipeline expects camera intrinsics[fx, fy, cx, cy]in original-image pixel coordinates; the resize-and-center-crop transform is applied internally.
Inference
import torch
from PIL import Image
from diffusers import SanaWMPipeline
from diffusers.utils import export_to_video
pipe = SanaWMPipeline.from_pretrained(
"Efficient-Large-Model/SANA-WM_bidirectional-diffusers",
torch_dtype=torch.bfloat16,
)
pipe.enable_model_cpu_offload() # ~45 GB of weights — offload between stages
# SANA-WM was trained on the LTX-2 VAE in framewise mode with tiling enabled. Without these
# settings the VAE encodes the whole (B, C, T, H, W) clip in one shot, which gives subtly
# different numerics from the released checkpoint.
pipe.vae.enable_tiling()
pipe.vae.use_framewise_encoding = True
pipe.vae.use_framewise_decoding = True
pipe.vae.tile_sample_stride_num_frames = 64
pipe.vae.tile_sample_min_num_frames = 96
prompt = "A car driving across a vast desert plain at golden hour."
output = pipe(
image=Image.open("input.png").convert("RGB"),
prompt=prompt,
action="w-80,jw-40,w-40", # WASD-style action DSL: forward 80f, jump+forward 40f, forward 40f
intrinsics=[800.0, 800.0, 845.0, 464.0], # fx, fy, cx, cy in original-image pixels
num_frames=161,
num_inference_steps=60,
guidance_scale=5.0,
generator=torch.Generator(device="cuda").manual_seed(42),
output_type="latent", # hand the latents to the refiner below
)
Pass action=None and supply your own c2w poses ((F, 4, 4) numpy array) to drive the camera trajectory
explicitly. Drop output_type="latent" to get video straight out of stage 1 and skip the refiner.
Stage 2 — the LTX-2 refiner
SanaWMLTX2Refiner ships as its own repository, the way SDXL splits base and refiner. Pass the base pipeline's VAE so the weights are shared rather than loaded twice:
from diffusers import SanaWMLTX2Refiner
refiner = SanaWMLTX2Refiner.from_pretrained(
"Efficient-Large-Model/SANA-WM_bidirectional-diffusers-refiner",
vae=pipe.vae,
torch_dtype=torch.bfloat16,
)
refiner.enable_model_cpu_offload()
frames = refiner(output.latent, prompt, fps=16)
export_to_video(list(frames), "sana_wm.mp4", fps=16)
Without a vae the refiner returns refined latents instead of video, which is useful if you want to decode
yourself.
Enable offloading on both pipelines, or free stage 1 before stage 2 (
pipe.transformer.to("cpu")). The two stages together are around 45 GB intorch.bfloat16, and keeping both resident on one 80 GB card leaves too little room for activations. Note also that stage 2 honourstorch_dtype: loading the refiner intorch.float32roughly doubles its memory and changes the output slightly.
If you don't have camera intrinsics, a hosted modular block can estimate them
from a single frame. It lives outside diffusers because it pulls in Pi3X — an extra dependency and a second
checkpoint — so nothing is downloaded until you ask for it:
from diffusers import ModularPipeline
# One-time per image. Requires `pip install pi3-vision`.
estimator = ModularPipeline.from_pretrained(
"Efficient-Large-Model/pi3x-intrinsics-estimator", trust_remote_code=True
)
estimator.load_components(dtype=torch.bfloat16)
intrinsics = estimator(image=Image.open("input.png").convert("RGB"), output="intrinsics")
Converting the released checkpoint
If you have the source SANA-WM release (not the pre-converted diffusers snapshot), run the conversion script once:
python scripts/convert_sana_wm_to_diffusers.py \
--src Efficient-Large-Model/SANA-WM_bidirectional \
--dst ./SANA-WM_bidirectional-diffusers
This writes two directories: the base pipeline at --dst, and the stage-2 refiner alongside it at
./SANA-WM_bidirectional-diffusers-refiner (override with --dst-refiner). Then load each from its local path
as usual.
Components
tokenizer—GemmaTokenizerFasttext_encoder— Gemma-2 (returns decoder hidden states)vae— AutoencoderKLLTX2Video (LTX-2, spatial ×32 / temporal ×8)transformer— SanaWMTransformer3DModel, 1.6B-parameter bidirectional DiTscheduler— FlowMatchEulerDiscreteScheduler
The stage-2 refiner is a separate repository with its own transformer
(SanaWMLTX2RefinerTransformer3DModel), connectors (LTX2TextConnectors), tokenizer, Gemma-3
text_encoder and scheduler. It has no vae of its own — pass the base pipeline's.
SanaWMPipeline[[diffusers.SanaWMPipeline]]
diffusers.SanaWMPipeline[[diffusers.SanaWMPipeline]]
diffusers.SanaWMPipeline(tokenizer: GemmaTokenizer | GemmaTokenizerFast, text_encoder: Gemma2PreTrainedModel, vae: AutoencoderKLLTX2Video, transformer: SanaWMTransformer3DModel, scheduler: FlowMatchEulerDiscreteScheduler)
Parameters:
tokenizer ([GemmaTokenizer] or [GemmaTokenizerFast]) : The Gemma-2 tokenizer.
text_encoder ([Gemma2PreTrainedModel]) : The Gemma-2 text encoder.
vae ([AutoencoderKLLTX2Video]) : The LTX-2 VAE.
transformer ([SanaWMTransformer3DModel]) : The 1600M bidirectional SANA-WM DiT.
scheduler ([FlowMatchEulerDiscreteScheduler]) : Flow-matching Euler scheduler (LTX-style per-token timesteps). latents directly.
SANA-WM camera-controlled image-to-video pipeline.
Generates a video from a first-frame image, a text prompt, and a camera trajectory (explicit c2w poses or a
WASD/IJKL action string). Uses the 1600M bidirectional SANA DiT for stage-1 sampling and the LTX-2
sink-bidirectional Euler refiner for stage-2 polish; both decode through the LTX-2 VAE.
call[[diffusers.SanaWMPipeline.call]]
__call__(image: PIL.Image.Image | str | Path, prompt: str, c2w: np.ndarray | None = None, action: str | None = None, intrinsics: np.ndarray | list[float] | None = None, height: int = 704, width: int = 1280, num_frames: int = 161, fps: int = 16, num_inference_steps: int = 60, guidance_scale: float = 5.0, negative_prompt: str = '', generator: torch.Generator | list[torch.Generator] | None = None, prompt_embeds: torch.Tensor | None = None, prompt_attention_mask: torch.Tensor | None = None, negative_prompt_embeds: torch.Tensor | None = None, negative_prompt_attention_mask: torch.Tensor | None = None, max_sequence_length: int = 300, chi_prompt: list[str] | None = None, output_type: Literal['np', 'pil', 'pt', 'latent'] = 'np', callback_on_step_end: Callable[[DiffusionPipeline, int, int, dict], dict] | None = None, callback_on_step_end_tensor_inputs: list[str] = ['latents'], return_dict: bool = True)
Parameters:
image (PIL.Image.Image or str) : First-frame image (PIL or path).
prompt (str) : Text prompt.
c2w (np.ndarray, optional) : (F, 4, 4) camera-to-world poses. Mutually exclusive with action.
action (str, optional) : Action-DSL string e.g. "w-80,jw-40,w-40". Mutually exclusive with c2w.
intrinsics (np.ndarray or list[float]) : [fx, fy, cx, cy] in original-image pixel coordinates. The pipeline applies the resize+crop transform internally.
height (int, defaults to 704) : Output frame height (fixed for the public model).
width (int, defaults to 1280) : Output frame width (fixed for the public model).
num_frames (int, defaults to 161) : Target frame count; snapped to 8k+1 (LTX-2 VAE constraint).
fps (int, defaults to 16) : Output frame rate (also fed to the refiner).
num_inference_steps (int, defaults to 60) : Number of stage-1 DiT sampling steps.
guidance_scale (float, defaults to 5.0) : Classifier-free guidance scale.
negative_prompt (str, defaults to "") : Optional negative prompt.
generator (torch.Generator or list[torch.Generator], optional) : One or more torch generators to make the noise sampling deterministic. If None, sampling is non-deterministic.
prompt_embeds (torch.Tensor, optional) : Pre-computed text embeddings, to skip the text encoder. Must be passed together with prompt_attention_mask.
prompt_attention_mask (torch.Tensor, optional) : Attention mask for prompt_embeds.
negative_prompt_embeds (torch.Tensor, optional) : Pre-computed negative text embeddings. Must be passed together with negative_prompt_attention_mask.
negative_prompt_attention_mask (torch.Tensor, optional) : Attention mask for negative_prompt_embeds.
max_sequence_length (int, defaults to 300) : Max prompt tokens.
chi_prompt (list[str], optional) : Override the chi-prompt prefix (default mirrors the public release).
output_type ("np", "pil", "pt", or "latent", defaults to "np") : Output format.
callback_on_step_end (Callable, optional) : Called at the end of each denoising step as callback_on_step_end(self, step, timestep, callback_kwargs), where callback_kwargs holds the tensors named in callback_on_step_end_tensor_inputs. Tensors in the returned dict replace the pipeline's.
callback_on_step_end_tensor_inputs (list[str], defaults to ["latents"]) : Which tensors to pass to callback_on_step_end; must be a subset of ._callback_tensor_inputs.
return_dict (bool, defaults to True) : Return [SanaWMPipelineOutput] vs tuple.
Returns:
[SanaWMPipelineOutput] with .frames of shape (T, H, W, 3), float np.ndarray in [0, 1] for
output_type="np", a list of PIL.Image.Image of length T for "pil", a (T, 3, H, W) tensor
for "pt", or the raw latent tensor for "latent".
Generate a SANA-WM camera-controlled video.
Examples:
>>> import torch
>>> from PIL import Image
>>> from diffusers import SanaWMPipeline
>>> pipe = SanaWMPipeline.from_pretrained(
... "Efficient-Large-Model/SANA-WM_bidirectional-diffusers", torch_dtype=torch.bfloat16
... ).to("cuda")
>>> output = pipe(
... image=Image.open("input.png").convert("RGB"),
... prompt="A car driving across a vast desert plain at golden hour.",
... action="w-80,jw-40,w-40",
... intrinsics=[800.0, 800.0, 845.0, 464.0], # fx, fy, cx, cy in original-image pixels
... num_inference_steps=60,
... )
>>> # output.frames is (T, H, W, 3) float np.ndarray in [0, 1] (diffusers convention).
check_inputs[[diffusers.SanaWMPipeline.check_inputs]]
check_inputs(c2w: np.ndarray | None, action: str | None, intrinsics: np.ndarray | list[float] | None, callback_on_step_end_tensor_inputs: list[str] | None = None)
Validate __call__ inputs. Raises on bad input and returns nothing.
encode_prompt[[diffusers.SanaWMPipeline.encode_prompt]]
encode_prompt(prompt: str, negative_prompt: str = '', device: torch.device | None = None, max_sequence_length: int = 300, chi_prompt: list[str] | None = None, prompt_embeds: torch.Tensor | None = None, prompt_attention_mask: torch.Tensor | None = None, negative_prompt_embeds: torch.Tensor | None = None, negative_prompt_attention_mask: torch.Tensor | None = None)
Returns:
(cond, cond_mask, neg, neg_mask) where cond and neg are (1, 1, L, D)-shaped Gemma hidden
states and the masks are (1, L).
Encode prompt + negative prompt through Gemma-2.
Mirrors the SANA chi-prompt-prefix trick: the chi prompt is prepended to the user prompt, then a select_index = [0, -L+1, ..., -1] slice takes the BOS token plus the last max_sequence_length - 1 tokens.
prepare_camera_trajectory[[diffusers.SanaWMPipeline.prepare_camera_trajectory]]
prepare_camera_trajectory(c2w: np.ndarray | None, action: str | None, intrinsics: np.ndarray | list[float], num_frames: int)
Normalize the camera inputs to (c2w_(F, 4, 4), intrinsics_(F, 4)).
Also snaps num_frames to the VAE-friendly 8k+1 and trims both arrays to match. The cropped image and
rescaled intrinsics come later, once the target resolution is known.
prepare_latents[[diffusers.SanaWMPipeline.prepare_latents]]
prepare_latents(first_latent: torch.Tensor, num_frames: int, height: int, width: int, dtype: torch.dtype, device: torch.device, generator: torch.Generator)
Sample initial latents and pin the first frame as the conditioning anchor.
Returns (latents, condition_mask) where condition_mask has ones on the first frame's tokens (they are
held clean throughout sampling) and zeros elsewhere.
SanaWMLTX2Refiner[[diffusers.SanaWMLTX2Refiner]]
The LTX-2 stage-2 refiner is a standalone DiffusionPipeline that takes stage-1 latents. Give it a vae (the
base pipeline's, so the weights are shared) to have it decode to video; without one it returns refined latents.
diffusers.SanaWMLTX2Refiner[[diffusers.SanaWMLTX2Refiner]]
diffusers.SanaWMLTX2Refiner(transformer: SanaWMLTX2RefinerTransformer3DModel, connectors: LTX2TextConnectors, tokenizer: GemmaTokenizer | GemmaTokenizerFast, text_encoder: Gemma3ForConditionalGeneration, scheduler: FlowMatchEulerDiscreteScheduler, vae: AutoencoderKLLTX2Video | None = None, text_max_sequence_length: int = 1024)
Parameters:
transformer ([SanaWMLTX2RefinerTransformer3DModel]) : The LTX-2 video DiT with the chunk-causal sliding-window KV cache.
connectors ([LTX2TextConnectors]) : LTX-2 text connectors.
tokenizer : Gemma-3 tokenizer.
text_encoder : Gemma-3 text encoder.
scheduler ([FlowMatchEulerDiscreteScheduler]) : Flow-matching Euler scheduler. Constructed with shift=1.0 so the distilled sigmas pass through unmodified.
vae ([AutoencoderKLLTX2Video], optional) : The same VAE used by [SanaWMPipeline]; pass vae=pipe.vae to share the weights. When given, the refiner decodes to video, otherwise it returns refined latents.
text_max_sequence_length (int, defaults to 1024) : Maximum tokens passed to the Gemma-3 tokenizer.
LTX-2 sink-bidirectional Euler refiner — SANA-WM stage 2, as a standalone pipeline.
Wraps the LTX-2 components (refiner transformer + text connectors + Gemma-3 text encoder + tokenizer) plus a [FlowMatchEulerDiscreteScheduler] that carries the distilled sigma schedule and performs the Euler steps. It is registered as an optional component of [SanaWMPipeline] and can also be used on its own to refine stage-1 latents.
call[[diffusers.SanaWMLTX2Refiner.call]]
__call__(sana_latent: torch.Tensor, prompt: str, fps: float, sink_size: int = 1, generator: torch.Generator | None = None, block_size: int = 3, kv_max_frames: int = 11, sigmas: tuple[float, ...] = (0.909375, 0.725, 0.421875, 0.0), output_type: str = 'np')
Parameters:
sana_latent : (B, C, F, H, W) stage-1 latent.
prompt : text prompt.
fps : video frame rate (drives LTX-2 RoPE temporal scaling).
sink_size : how many leading raw z_sana frames to anchor as the attention sink (canonical: 1).
generator : torch.Generator for the FM endpoint noise. Defaults to a generator seeded with 42 so results are reproducible out of the box.
block_size : latent frames per AR block (canonical: 3).
kv_max_frames : maximum context+active frames retained in the sliding window (canonical: 11 = 1 sink + 10 recent).
sigmas : descending Euler schedule terminating at 0.0 (canonical 3-step distilled: (0.909375, 0.725, 0.421875, 0.0)). Fed to self.scheduler (minus the trailing 0.0, which the scheduler appends itself).
output_type : "latent" returns the refined latents. Anything else decodes through self.vae and post-processes to that type ("np", "pt", "pil"); without a vae the latents are returned regardless.
Returns: *torch.Tensor*
Refined VAE latents of shape (B, C, F, H, W) — the first sink_size frames carry the
raw stage-1 sink latents unchanged, the rest carry the refined output.
Run the LTX-2 refiner and return refined VAE latents.
Uses the chunk-causal AR recipe the model was trained on (block_size=3, kv_max_frames=11): a sliding
window of [source_sink + recent_history + active_block] K/V is fed to the transformer one block at a time,
so per-block compute is bounded and total refinement cost scales linearly with video length.
SanaWMPipelineOutput[[diffusers.SanaWMPipelineOutput]]
diffusers.SanaWMPipelineOutput[[diffusers.SanaWMPipelineOutput]]
diffusers.SanaWMPipelineOutput(frames: typing.Union[torch.Tensor, numpy.ndarray, list[list[PIL.Image.Image]]], c2w: numpy.ndarray | None = None, latent: typing.Optional[torch.Tensor] = None)
Parameters:
frames (torch.Tensor, np.ndarray, or list[PIL.Image.Image]) : Generated video: a float np.ndarray of shape (T, H, W, 3) in [0, 1] for output_type="np", a (T, 3, H, W) tensor for "pt", a list of T ``PIL.Imagefor"pil", or the latent tensor (B, C, T_lat, H_lat, W_lat)for"latent"`.
c2w (np.ndarray) : Camera-to-world poses (T, 4, 4) aligned with frames (the refiner drops the sink anchor frame; this array is realigned accordingly when the refiner ran).
latent (torch.Tensor, optional) : Latent tensor in LTX-2 VAE space, shape (B, C, T_lat, H_lat, W_lat). Returned when `output_type="latent"``.
Output class for SANA-WM image-to-video pipeline.
Xet Storage Details
- Size:
- 20.6 kB
- Xet hash:
- 5a5b9729f7ae09d94e42eda3a3afd17be84729e864f198e0923574cc50baff90
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.