johndpope commited on
Commit
c12ff53
·
verified ·
1 Parent(s): 473f231

Fitted K/V up-maps (kv_maps/): sharp two-track video at 1.94x without a DiT fine-tune

Browse files
.gitattributes CHANGED
@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ kv_maps/before_after.png filter=lfs diff=lfs merge=lfs -text
37
+ kv_maps/quality_grid.png filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -14,7 +14,7 @@ tags:
14
  # MiniMax H3 two-track mixed resolution (MPEG-Diffusion)
15
 
16
  > [!WARNING]
17
- > **Research weights, help wanted.** The region head is published (`region_head/`). The LoRA is not published yet: its first run was stopped at step 500 to free the GPU for the region head. There is no image-quality result yet. Faces from the untrained sampler stay at half-resolution quality and can come out distorted. Training is meant to fix that. Help and results: [scrya-com/mpeg-diffusion issue #2](https://github.com/scrya-com/mpeg-diffusion/issues/2).
18
 
19
  > [!CAUTION]
20
  > **License and territory.** Derived from MiniMax H3 under the [MiniMax H3 Community License Agreement](LICENSE) (see [`NOTICE`](NOTICE)). Use and redistribution are allowed **only outside the European Union, the United Kingdom, the Republic of Korea and the United States of America**. Commercial use above US$20M yearly revenue needs MiniMax's written authorization. Do not use these weights to impersonate anyone, and disclose machine-generated output.
@@ -25,6 +25,8 @@ How it works: the unmodified H3 renders the whole frame at half resolution. A re
25
 
26
  | Folder | What |
27
  |---|---|
 
 
28
  | [`region_head/`](region_head) | The self-selected region head: one linear layer on block 25 of the frozen DiT that marks the subject partway through sampling, and so sets the full-resolution region and the Level-of-Token layout. Trained on 2,354 pictures with YOLOE subject labels; held-out mIoU 0.468. It favours the centre of the frame (see its card). |
29
  | LoRA (not published) | Lets the full-resolution track start earlier. First run (2,346 pairs, RTX 5090) stopped at step 500 with held-out loss down 21% (half resolution) and 26% (full resolution). |
30
 
 
14
  # MiniMax H3 two-track mixed resolution (MPEG-Diffusion)
15
 
16
  > [!WARNING]
17
+ > **Research weights, help wanted.** Working now, no DiT fine-tune: the fitted K/V maps ([`kv_maps/`](kv_maps)) and region head v3 ([`region_head_v3/`](region_head_v3)) give sharp subjects on 22-frame 1344×768 video at **1.94×** the speed of full resolution (RTX 5090). The LoRA is not published: its 500-step run changes the whole picture. Help and results: [scrya-com/mpeg-diffusion issue #2](https://github.com/scrya-com/mpeg-diffusion/issues/2).
18
 
19
  > [!CAUTION]
20
  > **License and territory.** Derived from MiniMax H3 under the [MiniMax H3 Community License Agreement](LICENSE) (see [`NOTICE`](NOTICE)). Use and redistribution are allowed **only outside the European Union, the United Kingdom, the Republic of Korea and the United States of America**. Commercial use above US$20M yearly revenue needs MiniMax's written authorization. Do not use these weights to impersonate anyone, and disclose machine-generated output.
 
25
 
26
  | Folder | What |
27
  |---|---|
28
+ | [`kv_maps/`](kv_maps) | Fitted `f_up` K/V maps (rank 128, 50 blocks). They remove the diamond-grid grain of copied keys in the full-resolution region. Use with `render --preset quality` or the ComfyUI node's `maps` input. |
29
+ | [`region_head_v3/`](region_head_v3) | Region head v3 (see its card); the one the quality recipe uses. |
30
  | [`region_head/`](region_head) | The self-selected region head: one linear layer on block 25 of the frozen DiT that marks the subject partway through sampling, and so sets the full-resolution region and the Level-of-Token layout. Trained on 2,354 pictures with YOLOE subject labels; held-out mIoU 0.468. It favours the centre of the frame (see its card). |
31
  | LoRA (not published) | Lets the full-resolution track start earlier. First run (2,346 pairs, RTX 5090) stopped at step 500 with held-out loss down 21% (half resolution) and 26% (full resolution). |
32
 
kv_maps/README.md ADDED
@@ -0,0 +1,62 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Fitted K/V up-maps (`f_up`) for two-track sampling on MiniMax H3
2
+
3
+ These maps let two-track sampling run without fine-tuning H3. Inside the full-resolution region, each query has to
4
+ read keys and values for the whole frame. Outside the region only half-resolution tokens exist, so each one has to
5
+ stand in for its 2×2 full-resolution cells.
6
+
7
+ Copying the same key to all 4 cells leaves a diamond-grid grain in the region, and more denoising steps do not
8
+ remove it. These maps add a fitted rank-128 correction per block, for keys and values, and the grain goes away. No
9
+ DiT weights change.
10
+
11
+ ![copied keys against fitted maps](before_after.png)
12
+
13
+ - **Method:** MPEG-Diffusion's dual-resolution attention: missing K/V come from linear maps applied before RoPE.
14
+ The paper is by Dennis Menn, Tianpei Gu, Guande He, Aryan V S, Yanke Song, Nan Liu, Desai Xie, Xun Huang,
15
+ Chenfeng Xu and Diana Marculescu (UT Austin and Roblox), [arXiv 2611.12069](https://arxiv.org/abs/2611.12069).
16
+ This is an independent port, not their release.
17
+ - **Fit:** `python3 lot/two_track_h3.py fit-kv` (ridge regression on the frozen H3, SVD-truncated to rank 128). It
18
+ uses still pairs at noise levels 0.2, 0.4, 0.6, 0.8 and 0.95. The median relative residual over the 50 blocks is:
19
+
20
+ | | Copy | Fitted |
21
+ |---|---|---|
22
+ | Keys | 0.62 | 0.52 |
23
+ | Values | 1.06 | 0.88 |
24
+ - **Format:** `up_maps_tt.safetensors`, float32, 32 MB. Keys `down.l{layer}_2x2_{k|v}` (128×128) and
25
+ `up.l{layer}_2x2_{k|v}` (128×512), for 50 layers. Metadata: `layers`, `rank`, `sigmas`.
26
+
27
+ ## Use
28
+
29
+ ```bash
30
+ git clone https://github.com/scrya-com/mpeg-diffusion && cd mpeg-diffusion
31
+ hf download johndpope/MiniMax-H3-MPEG-Diffusion kv_maps/up_maps_tt.safetensors region_head_v3/region_head_tt.safetensors --local-dir hf
32
+ export LOT_TT_MAPS=$PWD/hf/kv_maps/up_maps_tt.safetensors LOT_TT_REGION=$PWD/hf/region_head_v3/region_head_tt.safetensors
33
+ python3 lot/two_track_h3.py render --preset quality --frames 22 --out runs/q --prompts 4
34
+ ```
35
+
36
+ In ComfyUI ([ComfyUI-MiniMax-H3-Image-Lane](https://github.com/scrya-com/ComfyUI-MiniMax-H3-Image-Lane)), put the
37
+ file in `models/lot` and pick it as `maps` on the MiniMax H3 MPEG Diffusion node.
38
+
39
+ ## Result
40
+
41
+ The test: RTX 5090, 22 frames (7 latent frames) at 1344×768, 20 steps, switch 16, 6 HR steps, region head v3.
42
+ Decode is included, and no blocks are streamed.
43
+
44
+ | | Time |
45
+ |---|---|
46
+ | Full resolution | 27.2 s |
47
+ | Two-track with these maps | 14.0 s (**1.94×**) |
48
+
49
+ On 4 test prompts, the head picked 12–29% of the frame and two-track took 11.4–14.0 s against 25 s at full
50
+ resolution. Subjects are sharp, and there are no seams or flicker across frames.
51
+
52
+ ![full resolution against two-track](quality_grid.png)
53
+
54
+ Left: full resolution. Right: two-track, with the region outlined. They compose differently because they sample at
55
+ different resolutions. Middle frame of each clip.
56
+
57
+ Limits:
58
+ - Research weights.
59
+ - The value fit is weak (0.88 residual).
60
+ - The maps are fitted on stills; they work on these clips.
61
+
62
+ License: the [MiniMax H3 Community License](../LICENSE). Not for use in the EU, UK, Republic of Korea or USA.
kv_maps/before_after.png ADDED

Git LFS Details

  • SHA256: 0d1c8f42f43353d72fb4bb83236906ada765a55e086d6f9c7d139ecdb8d93f0f
  • Pointer size: 131 Bytes
  • Size of remote file: 681 kB
kv_maps/quality_grid.png ADDED

Git LFS Details

  • SHA256: 0d304b674407289d8def92cbd4cabed21b81bf6f2c95b705c2676932c931dcfa
  • Pointer size: 132 Bytes
  • Size of remote file: 2.49 MB
kv_maps/up_maps_tt.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a105eae0aa4f30d9a4cf1c861e7d78e8efd6393957ed6cfde09e34378984c648
3
+ size 32784896