Buckets:

hf-doc-build
/

doc

Files
xet
hf-doc-build/doc / diffusers /v0.43.0 /pt /api /pipelines /echo.html
mishig
2 days ago
download
raw
26.8 kB
<meta charset="utf-8" /><meta name="hf:doc:metadata" content="{&quot;title&quot;:&quot;Echo&quot;,&quot;local&quot;:&quot;echo&quot;,&quot;sections&quot;:[{&quot;title&quot;:&quot;Inferência&quot;,&quot;local&quot;:&quot;inference&quot;,&quot;sections&quot;:[],&quot;depth&quot;:2},{&quot;title&quot;:&quot;EchoModularPipeline&quot;,&quot;local&quot;:&quot;diffusers.EchoModularPipeline&quot;,&quot;sections&quot;:[],&quot;depth&quot;:2},{&quot;title&quot;:&quot;EchoBlocks&quot;,&quot;local&quot;:&quot;diffusers.EchoBlocks&quot;,&quot;sections&quot;:[],&quot;depth&quot;:2}],&quot;depth&quot;:1}"/>
		<link href="/docs/diffusers/v0.43.0/en/_app/immutable/entry/start.1gaXe03k.js" rel="modulepreload">
		<link href="/docs/diffusers/v0.43.0/en/_app/immutable/chunks/BObs6oQO.js" rel="modulepreload">
		<link href="/docs/diffusers/v0.43.0/en/_app/immutable/chunks/DPnyVmXs.js" rel="modulepreload">
		<link href="/docs/diffusers/v0.43.0/en/_app/immutable/entry/app.BwdYRbCn.js" rel="modulepreload">
		<link href="/docs/diffusers/v0.43.0/en/_app/immutable/chunks/DSGUT_OW.js" rel="modulepreload">
		<link href="/docs/diffusers/v0.43.0/en/_app/immutable/chunks/C045MWWx.js" rel="modulepreload">
		<link href="/docs/diffusers/v0.43.0/en/_app/immutable/chunks/6tuJ3nyi.js" rel="modulepreload">
		<link href="/docs/diffusers/v0.43.0/en/_app/immutable/chunks/DsnmJJEf.js" rel="modulepreload">
		<link href="/docs/diffusers/v0.43.0/en/_app/immutable/nodes/0.CornNeQR.js" rel="modulepreload">
		<link href="/docs/diffusers/v0.43.0/en/_app/immutable/chunks/IaDvMxLQ.js" rel="modulepreload">
		<link href="/docs/diffusers/v0.43.0/en/_app/immutable/nodes/177.B7YYsNYX.js" rel="modulepreload">
		<link href="/docs/diffusers/v0.43.0/en/_app/immutable/chunks/bKdwVYpU.js" rel="modulepreload">
		<!--rbrdnd--><meta name="hf:doc:metadata" content="{&quot;title&quot;:&quot;Echo&quot;,&quot;local&quot;:&quot;echo&quot;,&quot;sections&quot;:[{&quot;title&quot;:&quot;Inference&quot;,&quot;local&quot;:&quot;inference&quot;,&quot;sections&quot;:[],&quot;depth&quot;:2},{&quot;title&quot;:&quot;EchoModularPipeline&quot;,&quot;local&quot;:&quot;diffusers.EchoModularPipeline&quot;,&quot;sections&quot;:[],&quot;depth&quot;:2},{&quot;title&quot;:&quot;EchoBlocks&quot;,&quot;local&quot;:&quot;diffusers.EchoBlocks&quot;,&quot;sections&quot;:[],&quot;depth&quot;:2}],&quot;depth&quot;:1}"/><!---->
		<link href="/docs/diffusers/v0.43.0/en/_app/immutable/assets/0.tn0RQdqM.css" rel="modulepreload"> <!--[--><!--[0--><!--[--><!--[0--><!--[--><p></p> <!--[0--><h1 class="relative group"><a id="echo" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#echo"><span><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></span></a> <span><!---->Echo<!----></span></h1><!--]--><!----> <p><a href="https://github.com/jd-opensource/JoyAI-Echo" rel="nofollow">Echo</a> é um modelo de geração de vídeos longos. Ele suporta um primeiro frame limpo opcional, slots de memória de imagem/áudio ordenados e um amostrador estocástico de poucos passos de Distribution Matching Distillation (DMD).
O pipeline gera vídeo e áudio sincronizados.</p> <p>O Echo é implementado como um <a href="/docs/diffusers/v0.43.0/pt/api/modular_diffusers/pipeline#diffusers.ModularPipeline">ModularPipeline</a>, de modo que seus blocos de codificação de texto, condicionamento de memória, desruidificação estocática DMD e decodificação podem ser executados como um fluxo de trabalho completo ou compostos de forma independente.</p> <!--[1--><h2 class="relative group"><a id="inference" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#inference"><span><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></span></a> <span><!---->Inferência<!----></span></h2><!--]--><!----> <p>Carregue diretamente o checkpoint oficial <a href="https://huggingface.co/jdopensource/JoyAI-Echo" rel="nofollow">jdopensource/JoyAI-Echo</a>.
Ele usa o text encoder e o tokenizer de <a href="https://huggingface.co/google/gemma-3-12b-it" rel="nofollow">google/gemma-3-12b-it</a>.
O repositório Gemma é restrito, portanto, aceite sua licença e autentique-se no Hugging Face antes de carregar o pipeline.</p> <p>O modelo liberado usa 241 quadros em seu exemplo de vídeo longo. As coordenadas de RoPE do vídeo permanecem na taxa de treinamento de 24 fps, independentemente da taxa do contêiner de saída.</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5   text-gray-600  " title="code excerpt" type="button"><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg><!----> <div class=" absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0 "><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent;"></div> Copiado</div><!----></button><!----></div> <pre class="language-py "><!----><span class="hljs-keyword">import</span> torch
<span class="hljs-keyword">import</span> torchaudio
<span class="hljs-keyword">from</span> PIL <span class="hljs-keyword">import</span> Image

<span class="hljs-keyword">from</span> diffusers <span class="hljs-keyword">import</span> ComponentsManager, ModularPipeline
<span class="hljs-keyword">from</span> diffusers.utils <span class="hljs-keyword">import</span> encode_video

manager = ComponentsManager()
pipe = ModularPipeline.from_pretrained(<span class="hljs-string">&quot;jdopensource/JoyAI-Echo&quot;</span>, components_manager=manager)
pipe.load_components(dtype={<span class="hljs-string">&quot;default&quot;</span>: torch.bfloat16, <span class="hljs-string">&quot;audio_vae&quot;</span>: torch.float32})
manager.enable_auto_cpu_offload(device=<span class="hljs-string">&quot;cuda&quot;</span>)
pipe.vae.enable_tiling()

first_frame = Image.<span class="hljs-built_in">open</span>(<span class="hljs-string">&quot;first_frame.png&quot;</span>).convert(<span class="hljs-string">&quot;RGB&quot;</span>)
memory_images = [Image.<span class="hljs-built_in">open</span>(path).convert(<span class="hljs-string">&quot;RGB&quot;</span>) <span class="hljs-keyword">for</span> path <span class="hljs-keyword">in</span> [<span class="hljs-string">&quot;memory_0.png&quot;</span>, <span class="hljs-string">&quot;memory_1.png&quot;</span>]]
memory_audio_with_rates = [torchaudio.load(path) <span class="hljs-keyword">for</span> path <span class="hljs-keyword">in</span> [<span class="hljs-string">&quot;memory_0.wav&quot;</span>, <span class="hljs-string">&quot;memory_1.wav&quot;</span>]]
memory_audio = [waveform <span class="hljs-keyword">for</span> waveform, _ <span class="hljs-keyword">in</span> memory_audio_with_rates]
memory_audio_rates = [sample_rate <span class="hljs-keyword">for</span> _, sample_rate <span class="hljs-keyword">in</span> memory_audio_with_rates]

output = pipe(
    prompt=<span class="hljs-string">&quot;A cinematic dialogue scene in a quiet cafe.&quot;</span>,
    image=first_frame,
    memory_images=memory_images,
    memory_audio_waveforms=memory_audio,
    memory_audio_sample_rates=memory_audio_rates,
    width=<span class="hljs-number">1280</span>,
    height=<span class="hljs-number">736</span>,
    num_frames=<span class="hljs-number">241</span>,
    frame_rate=<span class="hljs-number">25.0</span>,
    model_frame_rate=<span class="hljs-number">24.0</span>,
    generator=torch.Generator(device=<span class="hljs-string">&quot;cuda&quot;</span>).manual_seed(<span class="hljs-number">42</span>),
    output_type=<span class="hljs-string">&quot;np&quot;</span>,
    output=[<span class="hljs-string">&quot;videos&quot;</span>, <span class="hljs-string">&quot;audio&quot;</span>],
)

encode_video(
    output[<span class="hljs-string">&quot;videos&quot;</span>][<span class="hljs-number">0</span>],
    fps=<span class="hljs-number">25</span>,
    audio=output[<span class="hljs-string">&quot;audio&quot;</span>][<span class="hljs-number">0</span>].<span class="hljs-built_in">float</span>().cpu(),
    audio_sample_rate=pipe.vocoder.config.output_sampling_rate,
    output_path=<span class="hljs-string">&quot;echo.mp4&quot;</span>,
)<!----></pre></div><!----> <p>O agendamento de sigma padrão do DMD é o agendamento de oito etapas publicado. Ele prevê <code>x0</code> em cada etapa e re-ruidifica com ruído gaussiano novo no próximo sigma, de modo que um <code>torch.Generator</code> com seed controla tanto o ruído inicial quanto todo o re-ruidificação intermediária.</p> <p>A codificação de áudio bruto em memória requer <code>torchaudio</code>. Para paridade de referência, mantenha <code>audio_vae</code> em FP32, conforme mostrado acima.
Fluxos de trabalho modulares podem fazer cache dos tensores normalizados e descompactados do codificador VAE. Os latentes de vídeo têm forma <code>(batch, channels, frames, height, width)</code> e os latentes de áudio têm forma <code>(batch, channels, time, mel_bins)</code>.
O bloco central <code>denoise</code> compacta esses tensores, expande o condicionamento para <code>num_videos_per_prompt</code> e descompacta
as saídas desruidadas de volta para a mesma forma do VAE. Passe os <code>latents</code> e <code>audio_latents</code> iniciais também nessa forma descompactada.
Os decodificadores recebem tensores VAE normalizados e os denormalizam imediatamente antes
da decodificação. <code>output=["latents", "audio_latents"]</code> retorna tensores VAE normalizados. Com <code>output_type="latent"</code> e <code>output=["videos", "audio"]</code>, você obtém os tensores VAE denormalizados sem decodificação.</p> <!--[1--><h2 class="relative group"><a id="diffusers.EchoModularPipeline" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#diffusers.EchoModularPipeline"><span><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></span></a> <span><!---->EchoModularPipeline<!----></span></h2><!--]--><!----> <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"><div><span class="group flex space-x-1.5 items-center text-gray-800 bg-gradient-to-r rounded-tr-lg -mt-4 -ml-4 pt-3 px-2.5" id="diffusers.EchoModularPipeline"><!----><h3 class="!m-0"><span class="flex-1 break-all md:text-lg bg-gradient-to-r px-2.5 py-1.5 rounded-xl from-indigo-50/70 to-white dark:from-gray-900 dark:to-gray-950 dark:text-indigo-300 text-indigo-700"><svg class="mr-1.5 text-indigo-500 inline-block -mt-0.5" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" focusable="false" role="img" width=".8em" height=".8em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 24 24"><path class="uim-quaternary" d="M20.23 7.24L12 12L3.77 7.24a1.98 1.98 0 0 1 .7-.71L11 2.76c.62-.35 1.38-.35 2 0l6.53 3.77c.29.173.531.418.7.71z" opacity=".25" fill="currentColor"></path><path class="uim-tertiary" d="M12 12v9.5a2.09 2.09 0 0 1-.91-.21L4.5 17.48a2.003 2.003 0 0 1-1-1.73v-7.5a2.06 2.06 0 0 1 .27-1.01L12 12z" opacity=".5" fill="currentColor"></path><path class="uim-primary" d="M20.5 8.25v7.5a2.003 2.003 0 0 1-1 1.73l-6.62 3.82c-.275.13-.576.198-.88.2V12l8.23-4.76c.175.308.268.656.27 1.01z" fill="currentColor"></path></svg><span class="font-light">classe</span> <span class="font-medium">diffusers.</span><span class="font-semibold">EchoModularPipeline</span></span></h3><!----> <a id="diffusers.EchoModularPipeline" class="header-link invisible with-hover:group-hover:visible pr-2" href="#diffusers.EchoModularPipeline"><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></a> <!--[0--><a class="!ml-auto !text-gray-400 !no-underline text-sm flex items-center" href="https://github.com/huggingface/diffusers/blob/v0.43.0/src/diffusers/modular_pipelines/echo/modular_pipeline.py#L27" target="_blank"><span>&lt;</span> <span class="hidden md:block mx-0.5 hover:!underline">source</span> <span>></span></a><!--]--></span> <!--[0--><p class="font-mono text-xs md:text-sm !leading-relaxed !my-6"><span>(</span> <!--[--><span class="comma cursor-default"><span class="rounded hover:bg-black hover:text-white dark:hover:bg-white dark:hover:text-black">blocks<span class="opacity-60">: diffusers.modular_pipelines.modular_pipeline.ModularPipelineBlocks | None = None</span></span></span><span class="comma cursor-default"><span class="rounded hover:bg-black hover:text-white dark:hover:bg-white dark:hover:text-black">pretrained_model_name_or_path<span class="opacity-60">: str | os.PathLike | None = None</span></span></span><span class="comma cursor-default"><span class="rounded hover:bg-black hover:text-white dark:hover:bg-white dark:hover:text-black">components_manager<span class="opacity-60">: diffusers.modular_pipelines.components_manager.ComponentsManager | None = None</span></span></span><span class="comma cursor-default"><span class="rounded hover:bg-black hover:text-white dark:hover:bg-white dark:hover:text-black">collection<span class="opacity-60">: str | None = None</span></span></span><span class="comma cursor-default"><span class="rounded hover:bg-black hover:text-white dark:hover:bg-white dark:hover:text-black">workflow<span class="opacity-60">: str | None = None</span></span></span><span class="comma cursor-default"><span class="rounded hover:bg-black hover:text-white dark:hover:bg-white dark:hover:text-black">modular_config_dict<span class="opacity-60">: dict[str, typing.Any] | None = None</span></span></span><span class="comma cursor-default"><span class="rounded hover:bg-black hover:text-white dark:hover:bg-white dark:hover:text-black">config_dict<span class="opacity-60">: dict[str, typing.Any] | None = None</span></span></span><span class="comma cursor-default"><span class="rounded hover:bg-black hover:text-white dark:hover:bg-white dark:hover:text-black">**kwargs<span class="opacity-60"></span></span></span><!--]--> <span>)</span> <!--[-1--><!--]--></p><!--]--> <div class="!mb-10 relative docstring-details "><!--[-1--><!--]--> <!--[-1--><!--]--> <!--[-1--><!--]--> <!--[-1--><!--]--> <!--[-1--><!--]--></div></div><!----> <p>Uma ModularPipeline para o checkpoint DMD de vídeo longo do Echo.</p></div> <!--[1--><h2 class="relative group"><a id="diffusers.EchoBlocks" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#diffusers.EchoBlocks"><span><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></span></a> <span><!---->EchoBlocks<!----></span></h2><!--]--><!----> <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"><div><span class="group flex space-x-1.5 items-center text-gray-800 bg-gradient-to-r rounded-tr-lg -mt-4 -ml-4 pt-3 px-2.5" id="diffusers.EchoBlocks"><!----><h3 class="!m-0"><span class="flex-1 break-all md:text-lg bg-gradient-to-r px-2.5 py-1.5 rounded-xl from-indigo-50/70 to-white dark:from-gray-900 dark:to-gray-950 dark:text-indigo-300 text-indigo-700"><svg class="mr-1.5 text-indigo-500 inline-block -mt-0.5" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" focusable="false" role="img" width=".8em" height=".8em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 24 24"><path class="uim-quaternary" d="M20.23 7.24L12 12L3.77 7.24a1.98 1.98 0 0 1 .7-.71L11 2.76c.62-.35 1.38-.35 2 0l6.53 3.77c.29.173.531.418.7.71z" opacity=".25" fill="currentColor"></path><path class="uim-tertiary" d="M12 12v9.5a2.09 2.09 0 0 1-.91-.21L4.5 17.48a2.003 2.003 0 0 1-1-1.73v-7.5a2.06 2.06 0 0 1 .27-1.01L12 12z" opacity=".5" fill="currentColor"></path><path class="uim-primary" d="M20.5 8.25v7.5a2.003 2.003 0 0 1-1 1.73l-6.62 3.82c-.275.13-.576.198-.88.2V12l8.23-4.76c.175.308.268.656.27 1.01z" fill="currentColor"></path></svg><span class="font-light">classe</span> <span class="font-medium">diffusers.</span><span class="font-semibold">EchoBlocks</span></span></h3><!----> <a id="diffusers.EchoBlocks" class="header-link invisible with-hover:group-hover:visible pr-2" href="#diffusers.EchoBlocks"><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg><!----></a> <!--[0--><a class="!ml-auto !text-gray-400 !no-underline text-sm flex items-center" href="https://github.com/huggingface/diffusers/blob/v0.43.0/src/diffusers/modular_pipelines/echo/modular_blocks_echo.py#L220" target="_blank"><span>&lt;</span> <span class="hidden md:block mx-0.5 hover:!underline">source</span> <span>></span></a><!--]--></span> <!--[0--><p class="font-mono text-xs md:text-sm !leading-relaxed !my-6"><span>(</span> <!--[--><!--]--> <span>)</span> <!--[-1--><!--]--></p><!--]--> <div class="!mb-10 relative docstring-details "><!--[-1--><!--]--> <!--[-1--><!--]--> <!--[-1--><!--]--> <!--[-1--><!--]--> <!--[-1--><!--]--></div></div><!----> <p>Geração de vídeo a partir de referência com Echo, condicionamento limpo do primeiro quadro, slots de memória de imagem/áudio ordenados e
desruidagem estocática DMD.</p> <p>Componentes:
text_encoder (<code>PreTrainedModel</code>) tokenizer (<code>PreTrainedTokenizerBase</code>) conectores (<code>LTX2TextConnectors</code>) vae
(<code>AutoencoderKLLTX2Video</code>) audio_vae (<code>AutoencoderKLLTX2Audio</code>) transformer (<code>LTX2VideoTransformer3DModel</code>)
video_processor (<code>VideoProcessor</code>) vocoder (<code>LTX2Vocoder</code>)</p> <p>Entradas:
prompt (<code>str</code>):
O prompt ou prompts para orientar a geração de imagem.
max_sequence_length (<code>int</code>, <em>opcional</em>, padrão 1024):
Comprimento máximo da sequência para a codificação do prompt.
image (<code>Image | Tensor</code>, <em>opcional</em>):
Primeiro quadro único opcional usado como condição de referência limpa.
memory_images (<code>list</code>, <em>opcional</em>):
Imagens de referência ordenadas, uma para cada slot de memória do Echo.
memory_audio_waveforms (<code>list</code>, <em>opcional</em>):
Formas de onda de memória ordenadas como tensores <code>(channels, samples)</code>. Entradas com mais de 9,62 segundos são recortadas para
a janela de maior resposta. Use <code>None</code> para um slot silencioso.
memory_audio_sample_rates (<code>int | list</code>, <em>opcional</em>):
Taxa de amostragem compartilhada por todas as formas de onda de memória, ou uma taxa por slot.
height (<code>int</code>, <em>opcional</em>, padrão 512):
A altura em pixels da imagem gerada.
width (<code>int</code>, <em>opcional</em>, padrão 704):
A largura em pixels da imagem gerada.
model_frame_rate (<code>float</code>, <em>opcional</em>, padrão 24.0):
Taxa de quadros no momento do treinamento usada para as coordenadas RoPE de vídeo do Echo.
memory_position_offset (<code>float</code>, <em>opcional</em>, padrão 500.0):
Centro temporal atribuído ao primeiro slot de memória.
memory_position_slot_stride (<code>float</code>, <em>opcional</em>, padrão 50.0):
Distância temporal entre os centros de slots de memória consecutivos.
num_videos_per_prompt (<code>int</code>, <em>opcional</em>, padrão 1):
Número de vídeos por prompt.
num_frames (<code>int</code>, <em>opcional</em>, padrão 241):
Número de quadros de pixels gerados; deve ser <code>1 + k * vae_temporal_compression_ratio</code>.
frame_rate (<code>float</code>, <em>opcional</em>, padrão 25.0):
Taxa de quadros do vídeo gerado.
latents (<code>Tensor</code>, <em>opcional</em>):
Ruído de vídeo inicial opcional em formato VAE (B, C, F, H, W).
audio_latents (<code>Tensor</code>, <em>opcional</em>):
Ruído de áudio inicial opcional em formato VAE (B, C, L, M).
generator (<code>Generator</code>, <em>opcional</em>):
Gerador do Torch para geração determinística.
sigmas (<code>list | tuple</code>):
Agendamento de sigma do DMD, incluindo o zero terminal.
attention_kwargs (<code>dict</code>, <em>opcional</em>):
Kwargs adicionais para os processadores de atenção.
output_type (<code>str</code>, <em>opcional</em>, padrão pil):
Formato de saída: ‘pil’, ‘np’, ‘pt’.
decode_timestep (<code>None</code>, <em>opcional</em>, padrão 0.0):
Timestep usado para decodificar os latents finais.
decode_noise_scale (<code>None</code>, <em>opcional</em>):
Fator de interpolação de ruído aplicado no timestep de decodificação.</p> <p>Saídas:
vídeos (<code>list</code>):
Os vídeos gerados.
áudio (<code>Tensor</code>):
A forma de onda de áudio gerada.</p></div> <a class="!text-gray-400 !no-underline text-sm flex items-center not-prose mt-4" href="https://github.com/huggingface/diffusers/blob/main/docs/source/en/api/pipelines/echo.md" target="_blank"><svg class="mr-1" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M31,16l-7,7l-1.41-1.41L28.17,16l-5.58-5.59L24,9l7,7z"></path><path d="M1,16l7-7l1.41,1.41L3.83,16l5.58,5.59L8,23l-7-7z"></path><path d="M12.419,25.484L17.639,6.552l1.932,0.518L14.351,26.002z"></path></svg><!----> <span><span class="underline">Update</span> on GitHub</span></a><!----> <p></p><!--]--><!----><!--]--><!--]--><!--]--> <!--[-1--><!--]--><!--]-->
			
			<script>
				{
					__sveltekit_ywq9g6 = {
						base: "/docs/diffusers/v0.43.0/en",
						assets: "/docs/diffusers/v0.43.0/en"
					};
					const element = document.currentScript.parentElement;
					Promise.all([
						import("/docs/diffusers/v0.43.0/en/_app/immutable/entry/start.1gaXe03k.js"),
						import("/docs/diffusers/v0.43.0/en/_app/immutable/entry/app.BwdYRbCn.js")
					]).then(([kit, app]) => {
						kit.start(app, element, {
							node_ids: [0, 177],
							data: [null,null],
							form: null,
							error: null
						});
					});
				}
			</script>
		
Xet Storage Details

Size:: 26.8 kB
Xet hash:: 97dbbc400a3ac8220e97d5bc52d58008ac9db8b06e3c1cf97cbb4fc260807880
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.