Buckets:

mishig's picture
|
download
raw
17.1 kB

원격 추론

이것은 현재 실험적 기능이며, 피드백이 있으시면 여기에 자유롭게 남겨 주세요.

Remote inference는 대형 모델의 로컬 추론 시 메모리 요구 사항을 완화하기 위해 디코딩 및 인코딩 과정을 원격 엔드포인트로 오프로드합니다. 이 기능은 Inference Endpoints에 의해 구동됩니다. 지원되는 모델과 엔드포인트는 아래 표를 참조하세요.

이 가이드에서는 원격 추론을 사용하여 latents를 인코딩하고 디코딩하는 방법을 보여줍니다.

인코딩

Encoding은 이미지와 비디오를 잠재 표현(latent representation)으로 변환합니다. 이 페이지 상단의 표는 인코딩을 지원하는 모델을 나열합니다.

이미지를 remote_encode()에 전달하여 인코딩합니다. 각 모델의 구체적인 scaling_factor 및 shift_factor 값은 Remote inference API 참고 문서를 참조하세요.

import torch
from diffusers import FluxPipeline
from diffusers.utils import load_image
from diffusers.utils.remote_utils import remote_encode

pipeline = FluxPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-schnell",
    dtype=torch.float16,
    vae=None,
    device_map="cuda"  # or "mps", "xpu", "cpu"
)

init_image = load_image(
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg"
)
init_image = init_image.resize((768, 512))

init_latent = remote_encode(
    endpoint="https://whhx50ex1aryqvw6.us-east-1.aws.endpoints.huggingface.cloud",
    image=init_image,
    scaling_factor=0.3611,
    shift_factor=0.1159
)

디코딩

Decoding은 잠재 표현을 다시 이미지나 비디오로 변환합니다. 이 페이지 상단의 표는 Decoding을 지원하는 모델을 나열합니다.

"latent"을 파이프라인의 출력 유형으로 설정하고 vae를 None으로 설정합니다. 이후 latents를 remote_decode() 함수에 전달합니다. Flux의 경우 latents가 패킹되어 있으므로 height와 width도 함께 전달해야 합니다. 각 모델에 대한 구체적인 scaling_factor 및 shift_factor 값은 Remote inference API 참고 문서를 참조하세요.

import torch
from diffusers import FluxPipeline
from diffusers.utils.remote_utils import remote_decode

pipeline = FluxPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-schnell",
    dtype=torch.bfloat16,
    vae=None,
    device_map="cuda"  # or "mps", "xpu", "cpu"
)

prompt = """
A photorealistic Apollo-era photograph of a cat in a small astronaut suit with a bubble helmet, standing on the Moon and holding a flagpole planted in the dusty lunar soil. The flag shows a colorful paw-print emblem. Earth glows in the black sky above the stark gray surface, with sharp shadows and high-contrast lighting like vintage NASA photos.
"""

latent = pipeline(
    prompt=prompt,
    guidance_scale=0.0,
    num_inference_steps=4,
    output_type="latent",
).images
image = remote_decode(
    endpoint="https://whhx50ex1aryqvw6.us-east-1.aws.endpoints.huggingface.cloud/",
    tensor=latent,
    height=1024,
    width=1024,
    scaling_factor=0.3611,
    shift_factor=0.1159,
)
image.save("image.jpg")
import torch
from diffusers import HunyuanVideoPipeline, HunyuanVideoTransformer3DModel
from diffusers.utils.remote_utils import remote_decode

transformer = HunyuanVideoTransformer3DModel.from_pretrained(
    "hunyuanvideo-community/HunyuanVideo", subfolder="transformer", dtype=torch.bfloat16
)
pipeline = HunyuanVideoPipeline.from_pretrained(
    "hunyuanvideo-community/HunyuanVideo", transformer=transformer, vae=None, dtype=torch.float16, device_map="cuda"  # or "mps", "xpu", "cpu"
)

latent = pipeline(
    prompt="A cat walks on the grass, realistic",
    height=320,
    width=512,
    num_frames=61,
    num_inference_steps=30,
    output_type="latent",
).frames

video = remote_decode(
    endpoint="https://o7ywnmrahorts457.us-east-1.aws.endpoints.huggingface.cloud/",
    tensor=latent,
    output_type="mp4",
    return_type="mp4",
)

with open("video.mp4", "wb") as f:
    f.write(video)

큐잉

Remote inference는 여러 생성 요청을 처리하기 위해 큐잉을 지원합니다. 현재 latent가 디코딩되는 동안 다음 프롬프트를 큐에 추가할 수 있습니다.

import queue
import threading
import torch
from IPython.display import display
from diffusers import StableDiffusionXLPipeline
from diffusers.utils.remote_utils import remote_decode

def decode_worker(q: queue.Queue):
    while True:
        item = q.get()
        if item is None:
            break
        image = remote_decode(
            endpoint="https://x2dmsqunjd6k9prw.us-east-1.aws.endpoints.huggingface.cloud/",
            tensor=item,
            scaling_factor=0.13025,
        )
        display(image)
        q.task_done()

q = queue.Queue()
thread = threading.Thread(target=decode_worker, args=(q,), daemon=True)
thread.start()

def decode(latent: torch.Tensor):
    q.put(latent)

prompts = [
    "A grainy Apollo-era style photograph of a cat in a snug astronaut suit with a bubble helmet, standing on the lunar surface and gripping a flag with a paw-print emblem. The gray Moon landscape stretches behind it, Earth glowing vividly in the black sky, shadows crisp and high-contrast.",
    "A vintage 1960s sci-fi pulp magazine cover illustration of a heroic cat astronaut planting a flag on the Moon. Bold, saturated colors, exaggerated space gear, playful typography floating in the background, Earth painted in bright blues and greens.",
    "A hyper-detailed cinematic shot of a cat astronaut on the Moon holding a fluttering flag, fur visible through the helmet glass, lunar dust scattering under its feet. The vastness of space and Earth in the distance create an epic, awe-inspiring tone.",
    "A colorful cartoon drawing of a happy cat wearing a chunky, oversized spacesuit, proudly holding a flag with a big paw print on it. The Moon’s surface is simplified with craters drawn like doodles, and Earth in the sky has a smiling face.",
    "A monochrome 1969-style press photo of a “first cat on the Moon” moment. The cat, in a tiny astronaut suit, stands by a planted flag, with grainy textures, scratches, and a blurred Earth in the background, mimicking old archival space photos."
]

pipeline = StableDiffusionXLPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    dtype=torch.float16,
    vae=None,
    device_map="cuda"  # or "mps", "xpu", "cpu"
)

pipeline.unet = pipeline.unet.to(memory_format=torch.channels_last)
pipeline.unet = torch.compile(pipeline.unet, mode="reduce-overhead", fullgraph=True)

_ = pipeline(
    prompt=prompts[0],
    output_type="latent",
)

for prompt in prompts:
    latent = pipeline(
        prompt=prompt,
        output_type="latent",
    ).images
    decode(latent)

q.put(None)
thread.join()

벤치마크

Stable Diffusion v1.5와 SDXL을 다양한 GPU에서 인코딩 및 디코딩할 때의 메모리 요구 사항을 보여줍니다.

이러한 GPU의 대부분에서 메모리 사용량이 다른 모델(텍스트 인코더, UNet/transformer)을 오프로드해야 하는지 또는 타일 인코딩이 필요한지를 결정합니다. 후자의 두 기법은 추론 시간을 늘리고 품질에 영향을 미칩니다.

인코딩 - Stable Diffusion v1.5

GPU 해결 시간(초) 메모리(%) 타일링 시간 (초) 타일 메모리(%)
NVIDIA GeForce RTX 4090 512x512 0.015 3.51901 0.015 3.51901
NVIDIA GeForce RTX 4090 256x256 0.004 1.3154 0.005 1.3154
NVIDIA GeForce RTX 4090 2048x2048 0.402 47.1852 0.496 3.51901
NVIDIA GeForce RTX 4090 1024x1024 0.078 12.2658 0.094 3.51901
NVIDIA GeForce RTX 4080 SUPER 512x512 0.023 5.30105 0.023 5.30105
NVIDIA GeForce RTX 4080 SUPER 256x256 0.006 1.98152 0.006 1.98152
NVIDIA GeForce RTX 4080 SUPER 2048x2048 0.574 71.08 0.656 5.30105
NVIDIA GeForce RTX 4080 SUPER 1024x1024 0.111 18.4772 0.14 5.30105
NVIDIA GeForce RTX 3090 512x512 0.032 3.52782 0.032 3.52782
NVIDIA GeForce RTX 3090 256x256 0.01 1.31869 0.009 1.31869
NVIDIA GeForce RTX 3090 2048x2048 0.742 47.3033 0.954 3.52782
NVIDIA GeForce RTX 3090 1024x1024 0.136 12.2965 0.207 3.52782
NVIDIA GeForce RTX 3080 512x512 0.036 8.51761 0.036 8.51761
NVIDIA GeForce RTX 3080 256x256 0.01 3.18387 0.01 3.18387
NVIDIA GeForce RTX 3080 2048x2048 0.863 86.7424 1.191 8.51761
NVIDIA GeForce RTX 3080 1024x1024 0.157 29.6888 0.227 8.51761
NVIDIA GeForce RTX 3070 512x512 0.051 10.6941 0.051 10.6941
NVIDIA GeForce RTX 3070 256x256 0.015 3.99743 0.015 3.99743
NVIDIA GeForce RTX 3070 2048x2048 1.217 96.054 1.482 10.6941
NVIDIA GeForce RTX 3070 1024x1024 0.223 37.2751 0.327 10.6941

SDXL 인코딩

GPU 해결 시간(초) 메모리 사용량(%) 타일링 시간 (초) 타일 메모리(%)
NVIDIA GeForce RTX 4090 512x512 0.029 4.95707 0.029 4.95707
NVIDIA GeForce RTX 4090 256x256 0.007 2.29666 0.007 2.29666
NVIDIA GeForce RTX 4090 2048x2048 0.873 66.3452 0.863 15.5649
NVIDIA GeForce RTX 4090 1024x1024 0.142 15.5479 0.143 15.5479
NVIDIA GeForce RTX 4080 SUPER 512x512 0.044 7.46735 0.044 7.46735
NVIDIA GeForce RTX 4080 SUPER 256x256 0.01 3.4597 0.01 3.4597
NVIDIA GeForce RTX 4080 SUPER 2048x2048 1.317 87.1615 1.291 23.447
NVIDIA GeForce RTX 4080 SUPER 1024x1024 0.213 23.4215 0.214 23.4215
NVIDIA GeForce RTX 3090 512x512 0.058 5.65638 0.058 5.65638
NVIDIA GeForce RTX 3090 256x256 0.016 2.45081 0.016 2.45081
NVIDIA GeForce RTX 3090 2048x2048 1.755 77.8239 1.614 18.4193
NVIDIA GeForce RTX 3090 1024x1024 0.265 18.4023 0.265 18.4023
NVIDIA GeForce RTX 3080 512x512 0.064 13.6568 0.064 13.6568
NVIDIA GeForce RTX 3080 256x256 0.018 5.91728 0.018 5.91728
NVIDIA GeForce RTX 3080 2048x2048 OOM OOM 1.866 44.4717
NVIDIA GeForce RTX 3080 1024x1024 0.302 44.4308 0.302 44.4308
NVIDIA GeForce RTX 3070 512x512 0.093 17.1465 0.093 17.1465
NVIDIA GeForce RTX 3070 256x256 0.025 7.42931 0.026 7.42931
NVIDIA GeForce RTX 3070 2048x2048 OOM OOM 2.674 55.8355
NVIDIA GeForce RTX 3070 1024x1024 0.443 55.7841 0.443 55.7841

Decoding - Stable Diffusion v1.5

GPU 해결 시간(초) 메모리(%) 타일링 시간 (초) 타일 메모리(%)
NVIDIA GeForce RTX 4090 512x512 0.031 5.60% 0.031 (0%) 5.60%
NVIDIA GeForce RTX 4090 1024x1024 0.148 20.00% 0.301 (+103%) 5.60%
NVIDIA GeForce RTX 4080 512x512 0.05 8.40% 0.050 (0%) 8.40%
NVIDIA GeForce RTX 4080 1024x1024 0.224 30.00% 0.356 (+59%) 8.40%
NVIDIA GeForce RTX 4070 Ti 512x512 0.066 11.30% 0.066 (0%) 11.30%
NVIDIA GeForce RTX 4070 Ti 1024x1024 0.284 40.50% 0.454 (+60%) 11.40%
NVIDIA GeForce RTX 3090 512x512 0.062 5.20% 0.062 (0%) 5.20%
NVIDIA GeForce RTX 3090 1024x1024 0.253 18.50% 0.464 (+83%) 5.20%
NVIDIA GeForce RTX 3080 512x512 0.07 12.80% 0.070 (0%) 12.80%
NVIDIA GeForce RTX 3080 1024x1024 0.286 45.30% 0.466 (+63%) 12.90%
NVIDIA GeForce RTX 3070 512x512 0.102 15.90% 0.102 (0%) 15.90%
NVIDIA GeForce RTX 3070 1024x1024 0.421 56.30% 0.746 (+77%) 16.00%

SDXL 디코딩

GPU 해결 시간(초) 메모리 사용량(%) 타일링 시간 (초) 타일 메모리(%)
NVIDIA GeForce RTX 4090 512x512 0.057 10.00% 0.057 (0%) 10.00%
NVIDIA GeForce RTX 4090 1024x1024 0.256 35.50% 0.257 (+0.4%) 35.50%
NVIDIA GeForce RTX 4080 512x512 0.092 15.00% 0.092 (0%) 15.00%
NVIDIA GeForce RTX 4080 1024x1024 0.406 53.30% 0.406 (0%) 53.30%
NVIDIA GeForce RTX 4070 Ti 512x512 0.121 20.20% 0.120 (-0.8%) 20.20%
NVIDIA GeForce RTX 4070 Ti 1024x1024 0.519 72.00% 0.519 (0%) 72.00%
NVIDIA GeForce RTX 3090 512x512 0.107 10.50% 0.107 (0%) 10.50%
NVIDIA GeForce RTX 3090 1024x1024 0.459 38.00% 0.460 (+0.2%) 38.00%
NVIDIA GeForce RTX 3080 512x512 0.121 25.60% 0.121 (0%) 25.60%
NVIDIA GeForce RTX 3080 1024x1024 0.524 93.00% 0.524 (0%) 93.00%
NVIDIA GeForce RTX 3070 512x512 0.183 31.80% 0.183 (0%) 31.80%
NVIDIA GeForce RTX 3070 1024x1024 0.794 96.40% 0.794 (0%) 96.40%

리소스

Xet Storage Details

Size:
17.1 kB
·
Xet hash:
c70568fd75c2c622ffd8f95715567aba45dd5bf347da0b3ba7a303fa3f5f0605

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.