Buckets:
| # Distributed inference | |
| Distributed inference splits the workload across multiple GPUs. It's useful for fitting larger models in memory and for processing more prompts at once. | |
| Diffusers supports four strategies. Each one has its own guide, and this page helps you pick one. | |
| ## Choosing a strategy | |
| Each strategy solves a different problem, so choose one based on the resource that limits you. | |
| | Strategy | Splits | Reduces | Latency for one prompt | Best when | | |
| |---|---|---|---|---| | |
| | [Data parallelism](../optimization/data_parallelism) | prompts across replicas | nothing, since each GPU holds a full copy | unchanged | the model already fits and you have many prompts | | |
| | [device_map](../optimization/device_map) | components across devices | weight memory | slightly worse | the model doesn't fit and the interconnect is slow | | |
| | [Context parallelism](../optimization/context_parallelism) | the input sequence | activation memory | lower | sequences are long, like high-resolution images or video | | |
| | [Tensor parallelism](../optimization/tensor_parallelism) | weight matrices | weight memory | lower | one component's weights don't fit and the interconnect is fast | | |
| Some practical guidance: | |
| - If the model already fits and you have many prompts, use data parallelism. It scales throughput with the number of GPUs without touching the model, and it leaves single-prompt latency alone. Set it up with [Accelerate](../optimization/data_parallelism#accelerate) or [PyTorch Distributed](../optimization/data_parallelism#pytorch-distributed). | |
| - If a single component's weights don't fit, try tensor parallelism first, since it lowers both memory and latency. Tensor parallelism communicates at every block boundary, so it needs a fast interconnect like NVLink. On PCIe, the per-layer traffic can outweigh the compute it saves, and `device_map` becomes the better choice. `device_map` also handles the case where the components are individually fine but collectively too large. | |
| - If activations, not weights, are the problem, use context parallelism. This helps with long sequences, like large images and videos with many frames. Pick a backend from the benchmarks in the [Context parallelism](../optimization/context_parallelism) guide. | |
| - If both the weights and the sequence are too large, Diffusers can't apply tensor and context parallelism to the same model. `ParallelConfig` raises an error if you pass both configs, and `enable_parallelism` can't be called a second time on a model that's already parallelized. | |
| The [xDiT](../optimization/xdit) and [ParaAttention](../optimization/para_attn) community integrations offer their own context parallelism implementations. | |
Xet Storage Details
- Size:
- 2.68 kB
- Xet hash:
- e2305baf9a5a9951650595e93123cec3e20519826452ded6118c3c33a5481da1
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.