Buckets:
Add tensor parallelism to a model
This guide is for contributors who want to add tensor parallelism support to a Diffusers model. To run a model that already supports it, refer to the Tensor parallelism guide.
Tensor parallelism works only on models that define a _tp_plan, a class attribute that maps module names to a sharding style. Writing one mostly means pairing each projection that expands the hidden dimension with the projection that contracts it back.
Each key can contain at most one *, and the part before the * must resolve to an nn.ModuleList, so one entry covers every block. A key without a * applies to the model itself. Paths are relative to the model.
Colwise and rowwise
| Style | Shards | Each rank | Use for |
|---|---|---|---|
"colwise" |
output features (weight dim 0) |
computes a slice of the output | to_q, to_k, to_v, FFN in-projection |
"rowwise" |
input features (weight dim 1) |
computes a partial sum | to_out.0, FFN out-projection |
Always pair them in that order. A "colwise" projection leaves its output sharded, the following "rowwise" projection consumes that shard directly, and a single AllReduce at the block boundary reconstructs the result. Sharding the pair any other way forces a gather in the middle and communicates far more.
In attention, the column-wise Q, K and V projections give each rank a subset of heads, so tp_degree must divide the head count. Encoder-stream duplicates (add_q_proj, to_add_out, ff_context) follow the same pattern as their image-stream counterparts.
Fused projections
When one Linear packs several logical tensors along the dimension being sharded, plain "colwise"/"rowwise" slices straight across the concatenation and misaligns the pieces. Use PackedColwiseParallel/PackedRowwiseParallel instead, which shard each packed block independently.
# in src/diffusers/models/transformers/your_model.py
from ...hooks.tensor_parallel import PackedColwiseParallel, PackedRowwiseParallel
Pass blocks, a list of proportional integers whose sum divides the packed dimension. For example, use [1, 1] for a SwiGLU gate and up projection of equal halves, or [1, 1, 1, 3, 3] for a fused Q, K, V, gate and up projection with mlp_ratio=3.
"transformer_blocks.*.ff.linear_in": PackedColwiseParallel([1, 1]),
When the block sizes are only known from the config, omit blocks and store the absolute sizes on the Linear during __init__ instead, as _tp_packed_col_blocks or _tp_packed_row_blocks.
# in the attention module's __init__
self.to_out._tp_packed_row_blocks = [self.inner_dim, self.mlp_hidden_dim]
# in the model's _tp_plan
"single_transformer_blocks.*.attn.to_out": PackedRowwiseParallel(),
The plan in transformer_flux2.py uses both forms. The double-stream SwiGLU pair passes blocks directly. The fused single-stream QKV and MLP projection stores its sizes on the Linear in Flux2ParallelSelfAttention.__init__, because they depend on mlp_ratio and mlp_mult_factor.
What to leave out
Modules left out of the plan are replicated on every rank. Leave out normalization layers, AdaLN modulation (img_mod and txt_mod), patch and text embeddings, and the final norm_out and proj_out. They're small, so sharding them saves little memory and adds communication.
Constraints and verification
tp_degreemust divideconfig.num_attention_heads.- Every packed block must individually be divisible by
tp_degree, not just their sum.
from_pretrained() and enable_parallelism() check both before loading or sharding any weight.
Validate a new plan numerically rather than by eye: generate with a fixed seed on a single device, then again under tensor parallelism, and compare the outputs. A misplaced "colwise"/"rowwise" usually still runs and produces a plausible but wrong image.
Start from an existing plan for a similar architecture.
transformer_qwenimage.pyhas no fused projections, so every entry is plain"colwise"or"rowwise".transformer_flux.pyadds one packed row-wise projection.transformer_flux2.pyuses both packed styles.
Xet Storage Details
- Size:
- 4.83 kB
- Xet hash:
- 95e8c6fdc0d6a83a437b2c4c58de565be8e25b261ddf637ae450873803ae956d
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.