multimodalart HF Staff commited on
Commit
caa8944
·
verified ·
1 Parent(s): 940839b

Fix garbled output: bind each transformer block to its own AoTI weights

Browse files

The Space currently returns pure noise for every generation (verified live: 30 steps, guidance 4.0, seed 42 on the model card's own example returns an 832x1152 image of RGB noise).

### Root cause: the AoTI graph runs all 40 blocks with block 0's weights

`package.pt2` is exported with the block's weights as *runtime* constants, so one compiled graph can serve all 40 `JoyImageEditPlusTransformerBlock`s — but each block still has to be bound to **its own** weights. The current code builds a single `ZeroGPUCompiledModel` from `_blocks[0].state_dict()` and applies that same instance to every block:

```python
_weights = ZeroGPUWeights(_blocks[0].state_dict(), to_cuda=True)
_compiled = ZeroGPUCompiledModel(_pt2_path, _weights)
for _blk in _blocks:
spaces.aoti_apply(_compiled, _blk)
```

`ZeroGPUCompiledModel.__call__` loads `self.weights.constants_map` once and reuses it, so the transformer effectively becomes block 0 repeated 40 times — no error, just noise.

The supported API does the right thing: given a `ModuleList`, `spaces.aoti_load_from_package_dir` patches every block with one shared `LazyAOTIModel` while keeping that block's own `state_dict` as the constants to load. (Same mechanism as `spaces.aoti_blocks_load`; used directly here because the artifact lives in a dataset repo.)

### Fallout: the demo was never actually resident on the GPU

`spaces.aoti_apply` calls `drain_module_parameters`, which replaced all 40 blocks' parameters with empty tensors — silently dropping ~29GB from the ZeroGPU pack. That is the only reason the Space fit the default 48GB MIG slice. With the weights correctly retained the pack is 50.3GB, which does not fit, so the GPU function now requests `size="xlarge"` (full 96GB card). Verified: without it the first CUDA op OOMs (`MIG 2g.48gb`, 47.4GiB).

### Two robustness fixes found along the way

- **`requirements.txt` pins `diffusers` to a moving branch.** The revision the Space is currently running is not the revision a rebuild would install. On the branch tip `JoyImageEditPlusPipeline` renamed `vae_image_processor` -> `image_processor` and changed the block's `image_rotary_emb` argument from `((cos, sin), None)` to `(cos, sin)` — the first breaks `app.py`, the second invalidates `package.pt2`. Pinned to `3fa516a2`, the branch tip the Space and the AoTI artifact were both built from.
- **`app.py` no longer calls `pipe.vae_image_processor`.** `height`/`width` are left unset; the pipeline derives the same 1024-base bucket from the last reference image internally, so this works on either pipeline revision.

### Verification

Tested end-to-end on a duplicate Space with the same inputs, prompt and seed as the `examples/` row. Before: RGB noise. After: a coherent generation matching the reference output.

*Alternative if `xlarge` (2x visitor quota) is not wanted:* drop the AoTI block and go back to `pipe.enable_model_cpu_offload()`. Peak VRAM then stays under the 48GB slice, at the cost of the compiled graph and per-call weight transfers. Happy to switch this PR to that shape instead.

Files changed (2) hide show
  1. app.py +30 -30
  2. requirements.txt +5 -1
app.py CHANGED
@@ -30,40 +30,39 @@ pipe.to("cuda")
30
  # AoTI: instead of compiling the repeated transformer block in this live demo
31
  # process, download the pre-compiled graph produced offline by the one-shot
32
  # Space multimodalart/joyai-image-edit-plus-aoti-export and published to the
33
- # public dataset below as ``package/root/package.pt2``. We load that compiled
34
- # graph and apply it to every ``JoyImageEditPlusTransformerBlock``
35
- # (``pipe.transformer.double_blocks``, 40 of them). Weights stay runtime inputs,
36
- # so the single compiled graph serves all blocks and no torch.compile runs here.
 
 
 
 
 
37
  #
38
  # Application happens at module scope (in the main process) so every forked
39
- # ZeroGPU worker inherits the patched blocks. Each block is wrapped in its own
40
- # ``ZeroGPUCompiledModel`` keyed by that block's weights (``ZeroGPUWeights``
41
- # moves the constants onto CUDA when the worker is forked, so cpu-offload's
42
- # CPU-resident weights are handled correctly). Falls back to eager execution if
43
  # the artifact can't be downloaded/applied.
44
  AOTI_DATASET_REPO = "multimodalart/joyai-image-edit-plus-aoti-pt2"
45
 
46
  try:
47
- from huggingface_hub import hf_hub_download
48
- from spaces.zero.torch.aoti import ZeroGPUCompiledModel, ZeroGPUWeights
 
49
 
50
- _pt2_path = hf_hub_download(
51
- repo_id=AOTI_DATASET_REPO,
52
- repo_type="dataset",
53
- filename="package/root/package.pt2",
 
 
 
 
 
54
  )
55
 
56
  _blocks = pipe.transformer.double_blocks
57
- # Build a single compiled model from the first block's weights and swap it
58
- # into every block — the same application the in-process compile did (which
59
- # applied one ``ZeroGPUCompiledModel`` to all blocks). to_cuda=True routes the
60
- # weights through ZeroGPU's fake-cuda pack machinery at module scope so the
61
- # constants are streamed onto real CUDA in the forked GPU worker (matching
62
- # what the compiled graph expects, even though the pipeline uses cpu-offload).
63
- _weights = ZeroGPUWeights(_blocks[0].state_dict(), to_cuda=True)
64
- _compiled = ZeroGPUCompiledModel(_pt2_path, _weights)
65
- for _blk in _blocks:
66
- spaces.aoti_apply(_compiled, _blk)
67
 
68
  print(
69
  f"AoTI: loaded compiled block graph from {AOTI_DATASET_REPO} and applied "
@@ -73,7 +72,10 @@ except Exception as e: # noqa: BLE001 -- keep the Space running if AoTI load fa
73
  print(f"AoTI load failed ({e!r}); running eager")
74
 
75
 
76
- @spaces.GPU(duration=300)
 
 
 
77
  def generate(
78
  images,
79
  prompt: str,
@@ -128,17 +130,15 @@ def generate(
128
  if randomize_seed:
129
  seed = random.randint(0, 2**31 - 1)
130
 
131
- # Determine output resolution from the last reference image
132
- target_h, target_w = pipe.vae_image_processor.get_default_height_width(pil_images[-1])
133
-
134
  generator = torch.Generator(device="cpu").manual_seed(int(seed))
135
 
 
 
 
136
  result = pipe(
137
  images=pil_images,
138
  prompt=prompt,
139
  negative_prompt=negative_prompt,
140
- height=target_h,
141
- width=target_w,
142
  num_inference_steps=int(num_inference_steps),
143
  guidance_scale=float(guidance_scale),
144
  generator=generator,
 
30
  # AoTI: instead of compiling the repeated transformer block in this live demo
31
  # process, download the pre-compiled graph produced offline by the one-shot
32
  # Space multimodalart/joyai-image-edit-plus-aoti-export and published to the
33
+ # public dataset below as ``package/root/package.pt2``.
34
+ #
35
+ # The graph was exported with the block's weights left as *runtime* constants,
36
+ # so the single compiled graph serves all 40
37
+ # ``JoyImageEditPlusTransformerBlock``s (``pipe.transformer.double_blocks``) --
38
+ # but each block must be bound to its OWN weights. ``aoti_load_from_package_dir``
39
+ # does exactly that: given a ``ModuleList`` it patches every block with the one
40
+ # shared ``LazyAOTIModel`` while keeping that block's own state_dict as the
41
+ # constants to load.
42
  #
43
  # Application happens at module scope (in the main process) so every forked
44
+ # ZeroGPU worker inherits the patched blocks. Falls back to eager execution if
 
 
 
45
  # the artifact can't be downloaded/applied.
46
  AOTI_DATASET_REPO = "multimodalart/joyai-image-edit-plus-aoti-pt2"
47
 
48
  try:
49
+ from pathlib import Path # noqa: E402
50
+
51
+ from huggingface_hub import snapshot_download
52
 
53
+ _package_dir = (
54
+ Path(
55
+ snapshot_download(
56
+ repo_id=AOTI_DATASET_REPO,
57
+ repo_type="dataset",
58
+ allow_patterns="package/*",
59
+ )
60
+ )
61
+ / "package"
62
  )
63
 
64
  _blocks = pipe.transformer.double_blocks
65
+ spaces.aoti_load_from_package_dir(_blocks, _package_dir)
 
 
 
 
 
 
 
 
 
66
 
67
  print(
68
  f"AoTI: loaded compiled block graph from {AOTI_DATASET_REPO} and applied "
 
72
  print(f"AoTI load failed ({e!r}); running eager")
73
 
74
 
75
+ # ``size="xlarge"`` (full 96GB card) is required, not optional: the pipeline is
76
+ # ~50GB of bf16 weights (16B transformer + Qwen3-VL-8B text encoder), which
77
+ # does not fit the default 48GB MIG slice.
78
+ @spaces.GPU(duration=300, size="xlarge")
79
  def generate(
80
  images,
81
  prompt: str,
 
130
  if randomize_seed:
131
  seed = random.randint(0, 2**31 - 1)
132
 
 
 
 
133
  generator = torch.Generator(device="cpu").manual_seed(int(seed))
134
 
135
+ # `height`/`width` are left unset: the pipeline picks the 1024-base bucket
136
+ # from the last reference image itself, which is what the model card
137
+ # recommends and avoids depending on the image-processor attribute name.
138
  result = pipe(
139
  images=pil_images,
140
  prompt=prompt,
141
  negative_prompt=negative_prompt,
 
 
142
  num_inference_steps=int(num_inference_steps),
143
  guidance_scale=float(guidance_scale),
144
  generator=generator,
requirements.txt CHANGED
@@ -1,4 +1,8 @@
1
- git+https://github.com/tangyanf/diffusers.git@add-joyimage-edit-plus
 
 
 
 
2
  transformers>=4.57.0
3
  accelerate
4
  sentencepiece
 
1
+ # Pinned to the exact branch commit this Space was built against: it is the
2
+ # revision the AoTI package.pt2 artifact was exported from, and later commits
3
+ # on the moving branch changed both the block's call signature and the
4
+ # pipeline's image-processor attribute name.
5
+ git+https://github.com/tangyanf/diffusers.git@3fa516a2201609cf0488b992dca96b148d512240
6
  transformers>=4.57.0
7
  accelerate
8
  sentencepiece