region_head_v3: region head (held-out mIoU 0.528)
Browse files
region_head_v3/README.md
ADDED
|
@@ -0,0 +1,67 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Region head: where the subject is, from MiniMax H3's own features
|
| 2 |
+
|
| 3 |
+
> [!WARNING]
|
| 4 |
+
> **Research checkpoint.** It finds the subject in most test prompts but not all. See *Limits* below.
|
| 5 |
+
> Help and results: [issue #5](https://github.com/scrya-com/mpeg-diffusion/issues/5) (which switch step) and [issue #2](https://github.com/scrya-com/mpeg-diffusion/issues/2) (the LoRA).
|
| 6 |
+
|
| 7 |
+
> [!CAUTION]
|
| 8 |
+
> Derived from MiniMax H3 under the [MiniMax H3 Community License](https://huggingface.co/johndpope/MiniMax-H3-MPEG-Diffusion/blob/main/LICENSE) (see [`NOTICE`](https://huggingface.co/johndpope/MiniMax-H3-MPEG-Diffusion/blob/main/NOTICE)):
|
| 9 |
+
> use only outside the European Union, the United Kingdom, the Republic of Korea and the United States of America.
|
| 10 |
+
|
| 11 |
+
The self-selected region of MPEG-Diffusion (Sec. 4.3) by Dennis Menn, Tianpei Gu, Guande He, Aryan V S, Yanke Song,
|
| 12 |
+
Nan Liu, Desai Xie, Xun Huang, Chenfeng Xu and Diana Marculescu (UT Austin and Roblox),
|
| 13 |
+
[arXiv 2611.12069](https://arxiv.org/abs/2611.12069), ported to MiniMax H3. Independent port, not their release.
|
| 14 |
+
|
| 15 |
+
`s = wᵀ LN(h)`: one linear layer (about 5,400 weights) on block 25's half-resolution video tokens,
|
| 16 |
+
partway through sampling. It scores each token; smoothing and the paper's threshold `m + α(p − m)` give the region,
|
| 17 |
+
and the region gives a Level-of-Token layout (full-size tokens inside, 2×2 on a ring, 4×4 elsewhere). The DiT is
|
| 18 |
+
frozen; nothing else is trained.
|
| 19 |
+
|
| 20 |
+
## Files
|
| 21 |
+
|
| 22 |
+
- `region_head_tt.safetensors`: the head, with `hidden` / `block` / `switch` metadata.
|
| 23 |
+
|
| 24 |
+
## Use
|
| 25 |
+
|
| 26 |
+
```bash
|
| 27 |
+
git clone https://github.com/scrya-com/mpeg-diffusion && cd mpeg-diffusion
|
| 28 |
+
hf download johndpope/MiniMax-H3-MPEG-Diffusion --include "region_head_v3/*" --local-dir weights
|
| 29 |
+
# Draw what it picks, and the LoT layout that implies, for held-out captions:
|
| 30 |
+
python3 lot/two_track_h3.py layout --region weights/region_head_v3/region_head_tt.safetensors --out out --switch 16
|
| 31 |
+
# Diffusion pass: LR for the switch, then the HR region for the remaining steps.
|
| 32 |
+
python3 lot/two_track_h3.py render --region weights/region_head_v3/region_head_tt.safetensors --out out \
|
| 33 |
+
--switch 16 --hr-steps 6 --outline
|
| 34 |
+
```
|
| 35 |
+
|
| 36 |
+
ComfyUI: [ComfyUI-MiniMax-H3-Image-Lane](https://github.com/scrya-com/ComfyUI-MiniMax-H3-Image-Lane).
|
| 37 |
+
Put this file in `ComfyUI/models/lot/` and load `workflows/mpeg_diffusion_api.json`. The node is
|
| 38 |
+
**MiniMax H3 MPEG Diffusion**. It runs the same pass on the base DiT (no two-track LoRA yet).
|
| 39 |
+
|
| 40 |
+
## Training
|
| 41 |
+
|
| 42 |
+
Data: the v2 labels (every person counts, no centrality score, 70% crop/flip) on the Pexels portraits and isometric stills, refit at switch steps 14, 16, 18 and 20 of 20. Best epoch 0, held-out mIoU 0.528. On H3's shift-12 schedule, step 16 (noise 0.80) is the earliest switch that finds an off-centre subject and still leaves high-resolution steps. Step 20 scores higher and leaves no high-resolution track. See https://github.com/scrya-com/mpeg-diffusion/issues/5
|
| 43 |
+
|
| 44 |
+
Phase 1: one forward of the frozen H3 per picture at half resolution, at one of the switch steps [14, 16, 18, 20] of
|
| 45 |
+
20 (cycled), keeping block 25's vector per token. 70% of pictures get a random
|
| 46 |
+
off-centre crop and a horizontal flip, so position does not predict the label. Phase 2: logistic regression on the
|
| 47 |
+
stored vectors; the epoch with the best held-out mIoU is kept. Code: `lot/train_tt.py region`
|
| 48 |
+
([how it works](https://github.com/scrya-com/mpeg-diffusion/blob/main/docs/REGION_HEAD.md)).
|
| 49 |
+
|
| 50 |
+
## Held-out mIoU (64 pictures)
|
| 51 |
+
|
| 52 |
+
Overall: **0.528** (the paper reports 0.59–0.67 against subject masks on Wan).
|
| 53 |
+
|
| 54 |
+
| Split | mIoU | Pictures |
|
| 55 |
+
|---|---|---|
|
| 56 |
+
| subject: other | 0.494 | 23 |
|
| 57 |
+
| subject: person | 0.547 | 41 |
|
| 58 |
+
| switch step 14 | 0.521 | 16 |
|
| 59 |
+
| switch step 16 | 0.471 | 16 |
|
| 60 |
+
| switch step 18 | 0.511 | 16 |
|
| 61 |
+
| switch step 20 | 0.610 | 16 |
|
| 62 |
+
|
| 63 |
+
## Limits
|
| 64 |
+
|
| 65 |
+
- The training data is mostly people (Pexels portraits and isometric scenes), so it is best at finding people.
|
| 66 |
+
- The held-out set has very few non-person subjects, so the "other" score is not reliable.
|
| 67 |
+
- Labels come from detectors (YuNet faces, YOLOE prompted with the caption's words), not from humans.
|
region_head_v3/region_head_tt.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:9a43b94c84b12c422cbf0d303c2145c87f0ed11d7ecd0e74b8bd5fa3a9715984
|
| 3 |
+
size 21780
|