johndpope commited on
Commit
473f231
·
verified ·
1 Parent(s): 324d39f

region_head_v3: region head (held-out mIoU 0.528)

Browse files
region_head_v3/README.md ADDED
@@ -0,0 +1,67 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Region head: where the subject is, from MiniMax H3's own features
2
+
3
+ > [!WARNING]
4
+ > **Research checkpoint.** It finds the subject in most test prompts but not all. See *Limits* below.
5
+ > Help and results: [issue #5](https://github.com/scrya-com/mpeg-diffusion/issues/5) (which switch step) and [issue #2](https://github.com/scrya-com/mpeg-diffusion/issues/2) (the LoRA).
6
+
7
+ > [!CAUTION]
8
+ > Derived from MiniMax H3 under the [MiniMax H3 Community License](https://huggingface.co/johndpope/MiniMax-H3-MPEG-Diffusion/blob/main/LICENSE) (see [`NOTICE`](https://huggingface.co/johndpope/MiniMax-H3-MPEG-Diffusion/blob/main/NOTICE)):
9
+ > use only outside the European Union, the United Kingdom, the Republic of Korea and the United States of America.
10
+
11
+ The self-selected region of MPEG-Diffusion (Sec. 4.3) by Dennis Menn, Tianpei Gu, Guande He, Aryan V S, Yanke Song,
12
+ Nan Liu, Desai Xie, Xun Huang, Chenfeng Xu and Diana Marculescu (UT Austin and Roblox),
13
+ [arXiv 2611.12069](https://arxiv.org/abs/2611.12069), ported to MiniMax H3. Independent port, not their release.
14
+
15
+ `s = wᵀ LN(h)`: one linear layer (about 5,400 weights) on block 25's half-resolution video tokens,
16
+ partway through sampling. It scores each token; smoothing and the paper's threshold `m + α(p − m)` give the region,
17
+ and the region gives a Level-of-Token layout (full-size tokens inside, 2×2 on a ring, 4×4 elsewhere). The DiT is
18
+ frozen; nothing else is trained.
19
+
20
+ ## Files
21
+
22
+ - `region_head_tt.safetensors`: the head, with `hidden` / `block` / `switch` metadata.
23
+
24
+ ## Use
25
+
26
+ ```bash
27
+ git clone https://github.com/scrya-com/mpeg-diffusion && cd mpeg-diffusion
28
+ hf download johndpope/MiniMax-H3-MPEG-Diffusion --include "region_head_v3/*" --local-dir weights
29
+ # Draw what it picks, and the LoT layout that implies, for held-out captions:
30
+ python3 lot/two_track_h3.py layout --region weights/region_head_v3/region_head_tt.safetensors --out out --switch 16
31
+ # Diffusion pass: LR for the switch, then the HR region for the remaining steps.
32
+ python3 lot/two_track_h3.py render --region weights/region_head_v3/region_head_tt.safetensors --out out \
33
+ --switch 16 --hr-steps 6 --outline
34
+ ```
35
+
36
+ ComfyUI: [ComfyUI-MiniMax-H3-Image-Lane](https://github.com/scrya-com/ComfyUI-MiniMax-H3-Image-Lane).
37
+ Put this file in `ComfyUI/models/lot/` and load `workflows/mpeg_diffusion_api.json`. The node is
38
+ **MiniMax H3 MPEG Diffusion**. It runs the same pass on the base DiT (no two-track LoRA yet).
39
+
40
+ ## Training
41
+
42
+ Data: the v2 labels (every person counts, no centrality score, 70% crop/flip) on the Pexels portraits and isometric stills, refit at switch steps 14, 16, 18 and 20 of 20. Best epoch 0, held-out mIoU 0.528. On H3's shift-12 schedule, step 16 (noise 0.80) is the earliest switch that finds an off-centre subject and still leaves high-resolution steps. Step 20 scores higher and leaves no high-resolution track. See https://github.com/scrya-com/mpeg-diffusion/issues/5
43
+
44
+ Phase 1: one forward of the frozen H3 per picture at half resolution, at one of the switch steps [14, 16, 18, 20] of
45
+ 20 (cycled), keeping block 25's vector per token. 70% of pictures get a random
46
+ off-centre crop and a horizontal flip, so position does not predict the label. Phase 2: logistic regression on the
47
+ stored vectors; the epoch with the best held-out mIoU is kept. Code: `lot/train_tt.py region`
48
+ ([how it works](https://github.com/scrya-com/mpeg-diffusion/blob/main/docs/REGION_HEAD.md)).
49
+
50
+ ## Held-out mIoU (64 pictures)
51
+
52
+ Overall: **0.528** (the paper reports 0.59–0.67 against subject masks on Wan).
53
+
54
+ | Split | mIoU | Pictures |
55
+ |---|---|---|
56
+ | subject: other | 0.494 | 23 |
57
+ | subject: person | 0.547 | 41 |
58
+ | switch step 14 | 0.521 | 16 |
59
+ | switch step 16 | 0.471 | 16 |
60
+ | switch step 18 | 0.511 | 16 |
61
+ | switch step 20 | 0.610 | 16 |
62
+
63
+ ## Limits
64
+
65
+ - The training data is mostly people (Pexels portraits and isometric scenes), so it is best at finding people.
66
+ - The held-out set has very few non-person subjects, so the "other" score is not reliable.
67
+ - Labels come from detectors (YuNet faces, YOLOE prompted with the caption's words), not from humans.
region_head_v3/region_head_tt.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9a43b94c84b12c422cbf0d303c2145c87f0ed11d7ecd0e74b8bd5fa3a9715984
3
+ size 21780