Spaces:
Running on Zero
Fix garbled output: bind each transformer block to its own AoTI weights
Browse filesThe Space currently returns pure noise for every generation (verified live: 30 steps, guidance 4.0, seed 42 on the model card's own example returns an 832x1152 image of RGB noise).
### Root cause: the AoTI graph runs all 40 blocks with block 0's weights
`package.pt2` is exported with the block's weights as *runtime* constants, so one compiled graph can serve all 40 `JoyImageEditPlusTransformerBlock`s — but each block still has to be bound to **its own** weights. The current code builds a single `ZeroGPUCompiledModel` from `_blocks[0].state_dict()` and applies that same instance to every block:
```python
_weights = ZeroGPUWeights(_blocks[0].state_dict(), to_cuda=True)
_compiled = ZeroGPUCompiledModel(_pt2_path, _weights)
for _blk in _blocks:
spaces.aoti_apply(_compiled, _blk)
```
`ZeroGPUCompiledModel.__call__` loads `self.weights.constants_map` once and reuses it, so the transformer effectively becomes block 0 repeated 40 times — no error, just noise.
The supported API does the right thing: given a `ModuleList`, `spaces.aoti_load_from_package_dir` patches every block with one shared `LazyAOTIModel` while keeping that block's own `state_dict` as the constants to load. (Same mechanism as `spaces.aoti_blocks_load`; used directly here because the artifact lives in a dataset repo.)
### Fallout: the demo was never actually resident on the GPU
`spaces.aoti_apply` calls `drain_module_parameters`, which replaced all 40 blocks' parameters with empty tensors — silently dropping ~29GB from the ZeroGPU pack. That is the only reason the Space fit the default 48GB MIG slice. With the weights correctly retained the pack is 50.3GB, which does not fit, so the GPU function now requests `size="xlarge"` (full 96GB card). Verified: without it the first CUDA op OOMs (`MIG 2g.48gb`, 47.4GiB).
### Two robustness fixes found along the way
- **`requirements.txt` pins `diffusers` to a moving branch.** The revision the Space is currently running is not the revision a rebuild would install. On the branch tip `JoyImageEditPlusPipeline` renamed `vae_image_processor` -> `image_processor` and changed the block's `image_rotary_emb` argument from `((cos, sin), None)` to `(cos, sin)` — the first breaks `app.py`, the second invalidates `package.pt2`. Pinned to `3fa516a2`, the branch tip the Space and the AoTI artifact were both built from.
- **`app.py` no longer calls `pipe.vae_image_processor`.** `height`/`width` are left unset; the pipeline derives the same 1024-base bucket from the last reference image internally, so this works on either pipeline revision.
### Verification
Tested end-to-end on a duplicate Space with the same inputs, prompt and seed as the `examples/` row. Before: RGB noise. After: a coherent generation matching the reference output.
*Alternative if `xlarge` (2x visitor quota) is not wanted:* drop the AoTI block and go back to `pipe.enable_model_cpu_offload()`. Peak VRAM then stays under the 48GB slice, at the cost of the compiled graph and per-call weight transfers. Happy to switch this PR to that shape instead.
- app.py +30 -30
- requirements.txt +5 -1
|
@@ -30,40 +30,39 @@ pipe.to("cuda")
|
|
| 30 |
# AoTI: instead of compiling the repeated transformer block in this live demo
|
| 31 |
# process, download the pre-compiled graph produced offline by the one-shot
|
| 32 |
# Space multimodalart/joyai-image-edit-plus-aoti-export and published to the
|
| 33 |
-
# public dataset below as ``package/root/package.pt2``.
|
| 34 |
-
#
|
| 35 |
-
#
|
| 36 |
-
# so the single compiled graph serves all
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 37 |
#
|
| 38 |
# Application happens at module scope (in the main process) so every forked
|
| 39 |
-
# ZeroGPU worker inherits the patched blocks.
|
| 40 |
-
# ``ZeroGPUCompiledModel`` keyed by that block's weights (``ZeroGPUWeights``
|
| 41 |
-
# moves the constants onto CUDA when the worker is forked, so cpu-offload's
|
| 42 |
-
# CPU-resident weights are handled correctly). Falls back to eager execution if
|
| 43 |
# the artifact can't be downloaded/applied.
|
| 44 |
AOTI_DATASET_REPO = "multimodalart/joyai-image-edit-plus-aoti-pt2"
|
| 45 |
|
| 46 |
try:
|
| 47 |
-
from
|
| 48 |
-
|
|
|
|
| 49 |
|
| 50 |
-
|
| 51 |
-
|
| 52 |
-
|
| 53 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 54 |
)
|
| 55 |
|
| 56 |
_blocks = pipe.transformer.double_blocks
|
| 57 |
-
|
| 58 |
-
# into every block — the same application the in-process compile did (which
|
| 59 |
-
# applied one ``ZeroGPUCompiledModel`` to all blocks). to_cuda=True routes the
|
| 60 |
-
# weights through ZeroGPU's fake-cuda pack machinery at module scope so the
|
| 61 |
-
# constants are streamed onto real CUDA in the forked GPU worker (matching
|
| 62 |
-
# what the compiled graph expects, even though the pipeline uses cpu-offload).
|
| 63 |
-
_weights = ZeroGPUWeights(_blocks[0].state_dict(), to_cuda=True)
|
| 64 |
-
_compiled = ZeroGPUCompiledModel(_pt2_path, _weights)
|
| 65 |
-
for _blk in _blocks:
|
| 66 |
-
spaces.aoti_apply(_compiled, _blk)
|
| 67 |
|
| 68 |
print(
|
| 69 |
f"AoTI: loaded compiled block graph from {AOTI_DATASET_REPO} and applied "
|
|
@@ -73,7 +72,10 @@ except Exception as e: # noqa: BLE001 -- keep the Space running if AoTI load fa
|
|
| 73 |
print(f"AoTI load failed ({e!r}); running eager")
|
| 74 |
|
| 75 |
|
| 76 |
-
|
|
|
|
|
|
|
|
|
|
| 77 |
def generate(
|
| 78 |
images,
|
| 79 |
prompt: str,
|
|
@@ -128,17 +130,15 @@ def generate(
|
|
| 128 |
if randomize_seed:
|
| 129 |
seed = random.randint(0, 2**31 - 1)
|
| 130 |
|
| 131 |
-
# Determine output resolution from the last reference image
|
| 132 |
-
target_h, target_w = pipe.vae_image_processor.get_default_height_width(pil_images[-1])
|
| 133 |
-
|
| 134 |
generator = torch.Generator(device="cpu").manual_seed(int(seed))
|
| 135 |
|
|
|
|
|
|
|
|
|
|
| 136 |
result = pipe(
|
| 137 |
images=pil_images,
|
| 138 |
prompt=prompt,
|
| 139 |
negative_prompt=negative_prompt,
|
| 140 |
-
height=target_h,
|
| 141 |
-
width=target_w,
|
| 142 |
num_inference_steps=int(num_inference_steps),
|
| 143 |
guidance_scale=float(guidance_scale),
|
| 144 |
generator=generator,
|
|
|
|
| 30 |
# AoTI: instead of compiling the repeated transformer block in this live demo
|
| 31 |
# process, download the pre-compiled graph produced offline by the one-shot
|
| 32 |
# Space multimodalart/joyai-image-edit-plus-aoti-export and published to the
|
| 33 |
+
# public dataset below as ``package/root/package.pt2``.
|
| 34 |
+
#
|
| 35 |
+
# The graph was exported with the block's weights left as *runtime* constants,
|
| 36 |
+
# so the single compiled graph serves all 40
|
| 37 |
+
# ``JoyImageEditPlusTransformerBlock``s (``pipe.transformer.double_blocks``) --
|
| 38 |
+
# but each block must be bound to its OWN weights. ``aoti_load_from_package_dir``
|
| 39 |
+
# does exactly that: given a ``ModuleList`` it patches every block with the one
|
| 40 |
+
# shared ``LazyAOTIModel`` while keeping that block's own state_dict as the
|
| 41 |
+
# constants to load.
|
| 42 |
#
|
| 43 |
# Application happens at module scope (in the main process) so every forked
|
| 44 |
+
# ZeroGPU worker inherits the patched blocks. Falls back to eager execution if
|
|
|
|
|
|
|
|
|
|
| 45 |
# the artifact can't be downloaded/applied.
|
| 46 |
AOTI_DATASET_REPO = "multimodalart/joyai-image-edit-plus-aoti-pt2"
|
| 47 |
|
| 48 |
try:
|
| 49 |
+
from pathlib import Path # noqa: E402
|
| 50 |
+
|
| 51 |
+
from huggingface_hub import snapshot_download
|
| 52 |
|
| 53 |
+
_package_dir = (
|
| 54 |
+
Path(
|
| 55 |
+
snapshot_download(
|
| 56 |
+
repo_id=AOTI_DATASET_REPO,
|
| 57 |
+
repo_type="dataset",
|
| 58 |
+
allow_patterns="package/*",
|
| 59 |
+
)
|
| 60 |
+
)
|
| 61 |
+
/ "package"
|
| 62 |
)
|
| 63 |
|
| 64 |
_blocks = pipe.transformer.double_blocks
|
| 65 |
+
spaces.aoti_load_from_package_dir(_blocks, _package_dir)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 66 |
|
| 67 |
print(
|
| 68 |
f"AoTI: loaded compiled block graph from {AOTI_DATASET_REPO} and applied "
|
|
|
|
| 72 |
print(f"AoTI load failed ({e!r}); running eager")
|
| 73 |
|
| 74 |
|
| 75 |
+
# ``size="xlarge"`` (full 96GB card) is required, not optional: the pipeline is
|
| 76 |
+
# ~50GB of bf16 weights (16B transformer + Qwen3-VL-8B text encoder), which
|
| 77 |
+
# does not fit the default 48GB MIG slice.
|
| 78 |
+
@spaces.GPU(duration=300, size="xlarge")
|
| 79 |
def generate(
|
| 80 |
images,
|
| 81 |
prompt: str,
|
|
|
|
| 130 |
if randomize_seed:
|
| 131 |
seed = random.randint(0, 2**31 - 1)
|
| 132 |
|
|
|
|
|
|
|
|
|
|
| 133 |
generator = torch.Generator(device="cpu").manual_seed(int(seed))
|
| 134 |
|
| 135 |
+
# `height`/`width` are left unset: the pipeline picks the 1024-base bucket
|
| 136 |
+
# from the last reference image itself, which is what the model card
|
| 137 |
+
# recommends and avoids depending on the image-processor attribute name.
|
| 138 |
result = pipe(
|
| 139 |
images=pil_images,
|
| 140 |
prompt=prompt,
|
| 141 |
negative_prompt=negative_prompt,
|
|
|
|
|
|
|
| 142 |
num_inference_steps=int(num_inference_steps),
|
| 143 |
guidance_scale=float(guidance_scale),
|
| 144 |
generator=generator,
|
|
@@ -1,4 +1,8 @@
|
|
| 1 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2 |
transformers>=4.57.0
|
| 3 |
accelerate
|
| 4 |
sentencepiece
|
|
|
|
| 1 |
+
# Pinned to the exact branch commit this Space was built against: it is the
|
| 2 |
+
# revision the AoTI package.pt2 artifact was exported from, and later commits
|
| 3 |
+
# on the moving branch changed both the block's call signature and the
|
| 4 |
+
# pipeline's image-processor attribute name.
|
| 5 |
+
git+https://github.com/tangyanf/diffusers.git@3fa516a2201609cf0488b992dca96b148d512240
|
| 6 |
transformers>=4.57.0
|
| 7 |
accelerate
|
| 8 |
sentencepiece
|