Buckets:

hf-doc-build/doc-dev / diffusers /pr_15172 /en /optimization /tensor_parallel_plan.md
|
download
raw
4.83 kB

Add tensor parallelism to a model

This guide is for contributors who want to add tensor parallelism support to a Diffusers model. To run a model that already supports it, refer to the Tensor parallelism guide.

Tensor parallelism works only on models that define a _tp_plan, a class attribute that maps module names to a sharding style. Writing one mostly means pairing each projection that expands the hidden dimension with the projection that contracts it back.

Each key can contain at most one *, and the part before the * must resolve to an nn.ModuleList, so one entry covers every block. A key without a * applies to the model itself. Paths are relative to the model.

Colwise and rowwise

Style Shards Each rank Use for
"colwise" output features (weight dim 0) computes a slice of the output to_q, to_k, to_v, FFN in-projection
"rowwise" input features (weight dim 1) computes a partial sum to_out.0, FFN out-projection

Always pair them in that order. A "colwise" projection leaves its output sharded, the following "rowwise" projection consumes that shard directly, and a single AllReduce at the block boundary reconstructs the result. Sharding the pair any other way forces a gather in the middle and communicates far more.

In attention, the column-wise Q, K and V projections give each rank a subset of heads, so tp_degree must divide the head count. Encoder-stream duplicates (add_q_proj, to_add_out, ff_context) follow the same pattern as their image-stream counterparts.

Fused projections

When one Linear packs several logical tensors along the dimension being sharded, plain "colwise"/"rowwise" slices straight across the concatenation and misaligns the pieces. Use PackedColwiseParallel/PackedRowwiseParallel instead, which shard each packed block independently.

# in src/diffusers/models/transformers/your_model.py
from ...hooks.tensor_parallel import PackedColwiseParallel, PackedRowwiseParallel

Pass blocks, a list of proportional integers whose sum divides the packed dimension. For example, use [1, 1] for a SwiGLU gate and up projection of equal halves, or [1, 1, 1, 3, 3] for a fused Q, K, V, gate and up projection with mlp_ratio=3.

"transformer_blocks.*.ff.linear_in": PackedColwiseParallel([1, 1]),

When the block sizes are only known from the config, omit blocks and store the absolute sizes on the Linear during __init__ instead, as _tp_packed_col_blocks or _tp_packed_row_blocks.

# in the attention module's __init__
self.to_out._tp_packed_row_blocks = [self.inner_dim, self.mlp_hidden_dim]
# in the model's _tp_plan
"single_transformer_blocks.*.attn.to_out": PackedRowwiseParallel(),

The plan in transformer_flux2.py uses both forms. The double-stream SwiGLU pair passes blocks directly. The fused single-stream QKV and MLP projection stores its sizes on the Linear in Flux2ParallelSelfAttention.__init__, because they depend on mlp_ratio and mlp_mult_factor.

What to leave out

Modules left out of the plan are replicated on every rank. Leave out normalization layers, AdaLN modulation (img_mod and txt_mod), patch and text embeddings, and the final norm_out and proj_out. They're small, so sharding them saves little memory and adds communication.

Constraints and verification

  • tp_degree must divide config.num_attention_heads.
  • Every packed block must individually be divisible by tp_degree, not just their sum.

from_pretrained() and enable_parallelism() check both before loading or sharding any weight.

Validate a new plan numerically rather than by eye: generate with a fixed seed on a single device, then again under tensor parallelism, and compare the outputs. A misplaced "colwise"/"rowwise" usually still runs and produces a plausible but wrong image.

Start from an existing plan for a similar architecture.

Xet Storage Details

Size:
4.83 kB
·
Xet hash:
95e8c6fdc0d6a83a437b2c4c58de565be8e25b261ddf637ae450873803ae956d

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.