Fitted K/V up-maps (kv_maps/): sharp two-track video at 1.94x without a DiT fine-tune
Browse files- .gitattributes +2 -0
- README.md +3 -1
- kv_maps/README.md +62 -0
- kv_maps/before_after.png +3 -0
- kv_maps/quality_grid.png +3 -0
- kv_maps/up_maps_tt.safetensors +3 -0
.gitattributes
CHANGED
|
@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
kv_maps/before_after.png filter=lfs diff=lfs merge=lfs -text
|
| 37 |
+
kv_maps/quality_grid.png filter=lfs diff=lfs merge=lfs -text
|
README.md
CHANGED
|
@@ -14,7 +14,7 @@ tags:
|
|
| 14 |
# MiniMax H3 two-track mixed resolution (MPEG-Diffusion)
|
| 15 |
|
| 16 |
> [!WARNING]
|
| 17 |
-
> **Research weights, help wanted.**
|
| 18 |
|
| 19 |
> [!CAUTION]
|
| 20 |
> **License and territory.** Derived from MiniMax H3 under the [MiniMax H3 Community License Agreement](LICENSE) (see [`NOTICE`](NOTICE)). Use and redistribution are allowed **only outside the European Union, the United Kingdom, the Republic of Korea and the United States of America**. Commercial use above US$20M yearly revenue needs MiniMax's written authorization. Do not use these weights to impersonate anyone, and disclose machine-generated output.
|
|
@@ -25,6 +25,8 @@ How it works: the unmodified H3 renders the whole frame at half resolution. A re
|
|
| 25 |
|
| 26 |
| Folder | What |
|
| 27 |
|---|---|
|
|
|
|
|
|
|
| 28 |
| [`region_head/`](region_head) | The self-selected region head: one linear layer on block 25 of the frozen DiT that marks the subject partway through sampling, and so sets the full-resolution region and the Level-of-Token layout. Trained on 2,354 pictures with YOLOE subject labels; held-out mIoU 0.468. It favours the centre of the frame (see its card). |
|
| 29 |
| LoRA (not published) | Lets the full-resolution track start earlier. First run (2,346 pairs, RTX 5090) stopped at step 500 with held-out loss down 21% (half resolution) and 26% (full resolution). |
|
| 30 |
|
|
|
|
| 14 |
# MiniMax H3 two-track mixed resolution (MPEG-Diffusion)
|
| 15 |
|
| 16 |
> [!WARNING]
|
| 17 |
+
> **Research weights, help wanted.** Working now, no DiT fine-tune: the fitted K/V maps ([`kv_maps/`](kv_maps)) and region head v3 ([`region_head_v3/`](region_head_v3)) give sharp subjects on 22-frame 1344×768 video at **1.94×** the speed of full resolution (RTX 5090). The LoRA is not published: its 500-step run changes the whole picture. Help and results: [scrya-com/mpeg-diffusion issue #2](https://github.com/scrya-com/mpeg-diffusion/issues/2).
|
| 18 |
|
| 19 |
> [!CAUTION]
|
| 20 |
> **License and territory.** Derived from MiniMax H3 under the [MiniMax H3 Community License Agreement](LICENSE) (see [`NOTICE`](NOTICE)). Use and redistribution are allowed **only outside the European Union, the United Kingdom, the Republic of Korea and the United States of America**. Commercial use above US$20M yearly revenue needs MiniMax's written authorization. Do not use these weights to impersonate anyone, and disclose machine-generated output.
|
|
|
|
| 25 |
|
| 26 |
| Folder | What |
|
| 27 |
|---|---|
|
| 28 |
+
| [`kv_maps/`](kv_maps) | Fitted `f_up` K/V maps (rank 128, 50 blocks). They remove the diamond-grid grain of copied keys in the full-resolution region. Use with `render --preset quality` or the ComfyUI node's `maps` input. |
|
| 29 |
+
| [`region_head_v3/`](region_head_v3) | Region head v3 (see its card); the one the quality recipe uses. |
|
| 30 |
| [`region_head/`](region_head) | The self-selected region head: one linear layer on block 25 of the frozen DiT that marks the subject partway through sampling, and so sets the full-resolution region and the Level-of-Token layout. Trained on 2,354 pictures with YOLOE subject labels; held-out mIoU 0.468. It favours the centre of the frame (see its card). |
|
| 31 |
| LoRA (not published) | Lets the full-resolution track start earlier. First run (2,346 pairs, RTX 5090) stopped at step 500 with held-out loss down 21% (half resolution) and 26% (full resolution). |
|
| 32 |
|
kv_maps/README.md
ADDED
|
@@ -0,0 +1,62 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Fitted K/V up-maps (`f_up`) for two-track sampling on MiniMax H3
|
| 2 |
+
|
| 3 |
+
These maps let two-track sampling run without fine-tuning H3. Inside the full-resolution region, each query has to
|
| 4 |
+
read keys and values for the whole frame. Outside the region only half-resolution tokens exist, so each one has to
|
| 5 |
+
stand in for its 2×2 full-resolution cells.
|
| 6 |
+
|
| 7 |
+
Copying the same key to all 4 cells leaves a diamond-grid grain in the region, and more denoising steps do not
|
| 8 |
+
remove it. These maps add a fitted rank-128 correction per block, for keys and values, and the grain goes away. No
|
| 9 |
+
DiT weights change.
|
| 10 |
+
|
| 11 |
+

|
| 12 |
+
|
| 13 |
+
- **Method:** MPEG-Diffusion's dual-resolution attention: missing K/V come from linear maps applied before RoPE.
|
| 14 |
+
The paper is by Dennis Menn, Tianpei Gu, Guande He, Aryan V S, Yanke Song, Nan Liu, Desai Xie, Xun Huang,
|
| 15 |
+
Chenfeng Xu and Diana Marculescu (UT Austin and Roblox), [arXiv 2611.12069](https://arxiv.org/abs/2611.12069).
|
| 16 |
+
This is an independent port, not their release.
|
| 17 |
+
- **Fit:** `python3 lot/two_track_h3.py fit-kv` (ridge regression on the frozen H3, SVD-truncated to rank 128). It
|
| 18 |
+
uses still pairs at noise levels 0.2, 0.4, 0.6, 0.8 and 0.95. The median relative residual over the 50 blocks is:
|
| 19 |
+
|
| 20 |
+
| | Copy | Fitted |
|
| 21 |
+
|---|---|---|
|
| 22 |
+
| Keys | 0.62 | 0.52 |
|
| 23 |
+
| Values | 1.06 | 0.88 |
|
| 24 |
+
- **Format:** `up_maps_tt.safetensors`, float32, 32 MB. Keys `down.l{layer}_2x2_{k|v}` (128×128) and
|
| 25 |
+
`up.l{layer}_2x2_{k|v}` (128×512), for 50 layers. Metadata: `layers`, `rank`, `sigmas`.
|
| 26 |
+
|
| 27 |
+
## Use
|
| 28 |
+
|
| 29 |
+
```bash
|
| 30 |
+
git clone https://github.com/scrya-com/mpeg-diffusion && cd mpeg-diffusion
|
| 31 |
+
hf download johndpope/MiniMax-H3-MPEG-Diffusion kv_maps/up_maps_tt.safetensors region_head_v3/region_head_tt.safetensors --local-dir hf
|
| 32 |
+
export LOT_TT_MAPS=$PWD/hf/kv_maps/up_maps_tt.safetensors LOT_TT_REGION=$PWD/hf/region_head_v3/region_head_tt.safetensors
|
| 33 |
+
python3 lot/two_track_h3.py render --preset quality --frames 22 --out runs/q --prompts 4
|
| 34 |
+
```
|
| 35 |
+
|
| 36 |
+
In ComfyUI ([ComfyUI-MiniMax-H3-Image-Lane](https://github.com/scrya-com/ComfyUI-MiniMax-H3-Image-Lane)), put the
|
| 37 |
+
file in `models/lot` and pick it as `maps` on the MiniMax H3 MPEG Diffusion node.
|
| 38 |
+
|
| 39 |
+
## Result
|
| 40 |
+
|
| 41 |
+
The test: RTX 5090, 22 frames (7 latent frames) at 1344×768, 20 steps, switch 16, 6 HR steps, region head v3.
|
| 42 |
+
Decode is included, and no blocks are streamed.
|
| 43 |
+
|
| 44 |
+
| | Time |
|
| 45 |
+
|---|---|
|
| 46 |
+
| Full resolution | 27.2 s |
|
| 47 |
+
| Two-track with these maps | 14.0 s (**1.94×**) |
|
| 48 |
+
|
| 49 |
+
On 4 test prompts, the head picked 12–29% of the frame and two-track took 11.4–14.0 s against 25 s at full
|
| 50 |
+
resolution. Subjects are sharp, and there are no seams or flicker across frames.
|
| 51 |
+
|
| 52 |
+

|
| 53 |
+
|
| 54 |
+
Left: full resolution. Right: two-track, with the region outlined. They compose differently because they sample at
|
| 55 |
+
different resolutions. Middle frame of each clip.
|
| 56 |
+
|
| 57 |
+
Limits:
|
| 58 |
+
- Research weights.
|
| 59 |
+
- The value fit is weak (0.88 residual).
|
| 60 |
+
- The maps are fitted on stills; they work on these clips.
|
| 61 |
+
|
| 62 |
+
License: the [MiniMax H3 Community License](../LICENSE). Not for use in the EU, UK, Republic of Korea or USA.
|
kv_maps/before_after.png
ADDED
|
Git LFS Details
|
kv_maps/quality_grid.png
ADDED
|
Git LFS Details
|
kv_maps/up_maps_tt.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:a105eae0aa4f30d9a4cf1c861e7d78e8efd6393957ed6cfde09e34378984c648
|
| 3 |
+
size 32784896
|