Add files using upload-large-folder tool

2d6b298 verified 11 days ago

8.48 kB

---
license: other
license_name: minimax-h3-community-license-agreement
license_link: LICENSE
base_model:
  - Comfy-Org/MiniMax-H3
  - ZhengmingYu/DMAD
base_model_relation: quantized
pipeline_tag: text-to-video
library_name: diffusion-single-file
tags:
  - minimax-h3
  - int8
  - convrot
  - dmad
  - distillation
  - few-step
  - text-to-audio-video
  - reference-to-video
---
# MiniMax-H3 FL2VA and Ref2VA (pruned) 4-step and 8-step DMAD: INT8 ConvRot

These are the pruned MiniMax-H3 FL2VA and Ref2VA transformers from [Comfy-Org/MiniMax-H3](https://huggingface.co/Comfy-Org/MiniMax-H3)
with a full-rank [DMAD](https://huggingface.co/ZhengmingYu/DMAD) LoRA merged in, quantized to INT8 ConvRot. The 4-step
files use the `full_critic` student and the 8-step files use `8step_full_critic`, which was trained for 8-step sampling.
All four were made the same way and differ only in the base model and the LoRA:

| File | Base model | Steps | Same layout as (Comfy-Org `diffusion_models/`) |
|---|---|---|---|
| `minimax_h3_fl2va_pruned_dmad_4step_int8_convrot.safetensors` | FL2VA | 4 | `minimax_h3_fl2va_pruned_int8_convrot.safetensors` |
| `minimax_h3_ref2va_pruned_dmad_4step_int8_convrot.safetensors` | Ref2VA | 4 | `minimax_h3_ref2va_pruned_int8_convrot.safetensors` |
| `minimax_h3_fl2va_pruned_dmad_8step_int8_convrot.safetensors` | FL2VA | 8 | `minimax_h3_fl2va_pruned_int8_convrot.safetensors` |
| `minimax_h3_ref2va_pruned_dmad_8step_int8_convrot.safetensors` | Ref2VA | 8 | `minimax_h3_ref2va_pruned_int8_convrot.safetensors` |

The tensor names and order, dtypes, shapes, data offsets and `.comfy_quant` descriptors all match the Comfy-Org file. The
difference is that the distillation is baked in, so you don't need a LoRA loader.

**Modified files:** the weights differ from the MiniMax-H3 release, as described under [How they were made](#how-they-were-made).

**Layout:** every block linear (`blocks.N.attn.{qkv_proj,out_proj}`, `blocks.N.mlp.{fc1,fc2}`) is INT8 and has its own
`.comfy_quant` tensor (`{"format": "int8_tensorwise", "convrot": true, "convrot_groupsize": 256}`) and a per-row F32
`weight_scale`. Everything else keeps the Comfy-Org dtype. 

**Not trained on Ref2VA:** DMAD was trained on the text-to-audio-video transformer, and the Ref2VA weights differ from
it. The DMAD authors state that the LoRAs load on the Ref2VA partition but don't publish results for it.

## Examples

### FL2VA, 4 steps

<video src="https://huggingface.co/Spectrallic/MiniMax-H3-DMAD-int8-convrot/resolve/main/samples/floating_city_1344x768_seed42.mp4" controls width="672"></video>

| Steps | CFG | Size | Frames | Seed | Sampler | Shift (video / audio) |
|---|---|---|---|---|---|---|
| 4 | 1.0 | 1344×768 | 124 at 24 fps | 42 | DMAD | 12 / 2 |

> A cinematic view of an ancient floating city suspended above a sea of cumulus clouds. The camera begins in a wide establishing shot and slowly dollies in toward the city, gliding forward through the soft pastel cloudscape as its intricate, hand-drawn architecture comes into view. A waterfall cascades from the city's edge into the clouds, while shafts of golden sunlight break through the cumulus clouds and illuminate the ultra-detailed digital painting.

This render used SageAttention with sparse attention at density 0.15 (first step dense) as described [here](https://noxengine.dev/blog/sparse/). 

### Ref2VA, 4 steps

<video src="https://huggingface.co/Spectrallic/MiniMax-H3-DMAD-int8-convrot/resolve/main/samples/stormy_boat_1344x768_seed42.mp4" controls width="672"></video>

| Steps | CFG | Size | Frames | Seed | Sampler | Shift (video / audio) |
|---|---|---|---|---|---|---|
| 4 | 1.0 | 1344×768 | 124 at 24 fps | 42 | DMAD | 12 / 2 |

> A close-up of a small wooden fishing boat in a stormy green sea at night, rain lashing the deck, while the floating city from &lt;Picture 1&gt; hangs in the sky far above and lightning flashes behind it. Handheld camera. Loud thunder and crashing waves.

`<Picture 1>` is the first frame of the FL2VA example above, scaled to match the output size. This render used the same setup as the FL2VA example: SageAttention with sparse attention at density 0.15 (first step dense).

### FL2VA, 8 steps

<video src="https://huggingface.co/Spectrallic/MiniMax-H3-DMAD-int8-convrot/resolve/main/samples/floating_city_8step_1344x768_seed42.mp4" controls width="672"></video>

| Steps | CFG | Size | Frames | Seed | Sampler | Shift (video / audio) |
|---|---|---|---|---|---|---|
| 8 | 1.0 | 1344×768 | 124 at 24 fps | 42 | DMAD | 12 / 2 |

Same prompt and render setup as the 4-step FL2VA example.

### Ref2VA, 8 steps

<video src="https://huggingface.co/Spectrallic/MiniMax-H3-DMAD-int8-convrot/resolve/main/samples/stormy_boat_8step_1344x768_seed42.mp4" controls width="672"></video>

| Steps | CFG | Size | Frames | Seed | Sampler | Shift (video / audio) |
|---|---|---|---|---|---|---|
| 8 | 1.0 | 1344×768 | 124 at 24 fps | 42 | DMAD | 12 / 2 |

Same prompt and render setup as the 4-step Ref2VA example. `<Picture 1>` is the first frame of the 8-step FL2VA example above.

## Sampling

Use each DMAD student's own operating point: **4 steps** for the 4-step files and **8 steps** for the 8-step files,
CFG 1 (no classifier-free guidance), video time shift 12 and audio shift 2. Workflows are in the [DMAD repository](https://huggingface.co/ZhengmingYu/DMAD).

Both examples were rendered in Nox with the DMAD sampler. For Ref2VA, refer to references in the prompt as `<Picture i>`,
`<Video k>` and `<Audio j>`.

## How they were made

`<variant>` below is `fl2va` or `ref2va`, and `<steps>` is `4step` or `8step`.

1. **Source:**
   - Base: `diffusion_models/minimax_h3_<variant>_pruned_bf16.safetensors` from Comfy-Org/MiniMax-H3 (Ref2VA: revision
     `912b6e38`).
   - LoRA: `minimax_h3/dmad_minimax_h3_<steps>_full_critic_comfyui.safetensors` from ZhengmingYu/DMAD (revision
     `b08c2bf3`). These are the authors' exact conversions of `full_critic` and `8step_full_critic`, with no rank
     reduction: rank 128, with q/k/v fused into `attn.qkv_proj` adapters of rank 384.
   - Layout reference: `diffusion_models/minimax_h3_<variant>_pruned_int8_convrot.safetensors` from Comfy-Org/MiniMax-H3
     (FL2VA: revision `d6cfb4e5`; Ref2VA: revision `912b6e38`).
2. **Merge:** the LoRA is merged at strength **1.0** into its 208 targets: `attn.{qkv_proj,out_proj}` and `mlp.{fc1,fc2}`
   in all 50 blocks and both `token_refiner` blocks. Each merge computes `W' = W + (alpha/rank) · up @ down` in FP32.
3. **Quantize:** the 200 block linears are rotated with a group-256 Hadamard ConvRot, then quantized to INT8 with
   per-row scales, directly from the FP32 merge result. The 8 merged `token_refiner` weights go back to BF16, because
   that layer stays unquantized in the Comfy-Org file.
4. **Copy everything else:** tensors the LoRA doesn't touch are copied bit for bit from the layout reference.
### Checks

- **Against the Comfy-Org INT8 file, byte for byte (all four files):** all 932 tensors match in name, order, dtype, shape and offset. Only
  408 tensors have different bytes: the 200 INT8 weights, their 200 scales and the 8 `token_refiner` weights. The other
  524 are identical, including all 200 `.comfy_quant` tensors.
- **LoRA presence:** this projects the change from the base onto the LoRA delta, where 1 means fully
  present. 4 steps: the overall result is 0.9977. Of the 208 layers, 205 are within 1 ± 0.1. The other three are
  `token_refiner.blocks.0.mlp.fc2` (0.647), `token_refiner.blocks.1.attn.qkv_proj` (0.896) and
  `token_refiner.blocks.1.attn.out_proj` (0.899).
  8 steps: the overall result is 0.9912 for FL2VA and 0.9913 for Ref2VA. All 200 block layers are within 1 ± 0.1. The 8
  `token_refiner` layers fall between 0.36 and 0.73: their 8-step delta is only about 0.07% of the weight, which is below
  BF16 rounding, so part of it rounds away when they are stored in their Comfy-Org dtype.

## Credits & license

- Base model: [MiniMaxAI/MiniMax-H3](https://huggingface.co/MiniMaxAI/MiniMax-H3), repackaged by
  [Comfy-Org/MiniMax-H3](https://huggingface.co/Comfy-Org/MiniMax-H3)
- Distillation: [ZhengmingYu/DMAD](https://huggingface.co/ZhengmingYu/DMAD)
  ([paper](https://arxiv.org/abs/2610.02188), [code](https://github.com/Yzmblog/DMAD))

This derivative is released under the [MiniMax H3 Community License Agreement](LICENSE), which includes territorial
restrictions. See `LICENSE` and `NOTICE`.

Powered by MiniMax H3.

	---
	license: other
	license_name: minimax-h3-community-license-agreement
	license_link: LICENSE
	base_model:
	- Comfy-Org/MiniMax-H3
	- ZhengmingYu/DMAD
	base_model_relation: quantized
	pipeline_tag: text-to-video
	library_name: diffusion-single-file
	tags:
	- minimax-h3
	- int8
	- convrot
	- dmad
	- distillation
	- few-step
	- text-to-audio-video
	- reference-to-video
	---

	# MiniMax-H3 FL2VA and Ref2VA (pruned) 4-step and 8-step DMAD: INT8 ConvRot

	These are the pruned MiniMax-H3 FL2VA and Ref2VA transformers from [Comfy-Org/MiniMax-H3](https://huggingface.co/Comfy-Org/MiniMax-H3)
	with a full-rank [DMAD](https://huggingface.co/ZhengmingYu/DMAD) LoRA merged in, quantized to INT8 ConvRot. The 4-step
	files use the `full_critic` student and the 8-step files use `8step_full_critic`, which was trained for 8-step sampling.
	All four were made the same way and differ only in the base model and the LoRA:

	\| File \| Base model \| Steps \| Same layout as (Comfy-Org `diffusion_models/`) \|
	\|---\|---\|---\|---\|
	\| `minimax_h3_fl2va_pruned_dmad_4step_int8_convrot.safetensors` \| FL2VA \| 4 \| `minimax_h3_fl2va_pruned_int8_convrot.safetensors` \|
	\| `minimax_h3_ref2va_pruned_dmad_4step_int8_convrot.safetensors` \| Ref2VA \| 4 \| `minimax_h3_ref2va_pruned_int8_convrot.safetensors` \|
	\| `minimax_h3_fl2va_pruned_dmad_8step_int8_convrot.safetensors` \| FL2VA \| 8 \| `minimax_h3_fl2va_pruned_int8_convrot.safetensors` \|
	\| `minimax_h3_ref2va_pruned_dmad_8step_int8_convrot.safetensors` \| Ref2VA \| 8 \| `minimax_h3_ref2va_pruned_int8_convrot.safetensors` \|

	The tensor names and order, dtypes, shapes, data offsets and `.comfy_quant` descriptors all match the Comfy-Org file. The
	difference is that the distillation is baked in, so you don't need a LoRA loader.

	Modified files: the weights differ from the MiniMax-H3 release, as described under [How they were made](#how-they-were-made).

	Layout: every block linear (`blocks.N.attn.{qkv_proj,out_proj}`, `blocks.N.mlp.{fc1,fc2}`) is INT8 and has its own
	`.comfy_quant` tensor (`{"format": "int8_tensorwise", "convrot": true, "convrot_groupsize": 256}`) and a per-row F32
	`weight_scale`. Everything else keeps the Comfy-Org dtype.

	Not trained on Ref2VA: DMAD was trained on the text-to-audio-video transformer, and the Ref2VA weights differ from
	it. The DMAD authors state that the LoRAs load on the Ref2VA partition but don't publish results for it.

	## Examples

	### FL2VA, 4 steps

	<video src="https://huggingface.co/Spectrallic/MiniMax-H3-DMAD-int8-convrot/resolve/main/samples/floating_city_1344x768_seed42.mp4" controls width="672"></video>

	\| Steps \| CFG \| Size \| Frames \| Seed \| Sampler \| Shift (video / audio) \|
	\|---\|---\|---\|---\|---\|---\|---\|
	\| 4 \| 1.0 \| 1344×768 \| 124 at 24 fps \| 42 \| DMAD \| 12 / 2 \|

	> A cinematic view of an ancient floating city suspended above a sea of cumulus clouds. The camera begins in a wide establishing shot and slowly dollies in toward the city, gliding forward through the soft pastel cloudscape as its intricate, hand-drawn architecture comes into view. A waterfall cascades from the city's edge into the clouds, while shafts of golden sunlight break through the cumulus clouds and illuminate the ultra-detailed digital painting.

	This render used SageAttention with sparse attention at density 0.15 (first step dense) as described [here](https://noxengine.dev/blog/sparse/).

	### Ref2VA, 4 steps

	<video src="https://huggingface.co/Spectrallic/MiniMax-H3-DMAD-int8-convrot/resolve/main/samples/stormy_boat_1344x768_seed42.mp4" controls width="672"></video>

	\| Steps \| CFG \| Size \| Frames \| Seed \| Sampler \| Shift (video / audio) \|
	\|---\|---\|---\|---\|---\|---\|---\|
	\| 4 \| 1.0 \| 1344×768 \| 124 at 24 fps \| 42 \| DMAD \| 12 / 2 \|

	> A close-up of a small wooden fishing boat in a stormy green sea at night, rain lashing the deck, while the floating city from <Picture 1> hangs in the sky far above and lightning flashes behind it. Handheld camera. Loud thunder and crashing waves.

	`<Picture 1>` is the first frame of the FL2VA example above, scaled to match the output size. This render used the same setup as the FL2VA example: SageAttention with sparse attention at density 0.15 (first step dense).

	### FL2VA, 8 steps

	<video src="https://huggingface.co/Spectrallic/MiniMax-H3-DMAD-int8-convrot/resolve/main/samples/floating_city_8step_1344x768_seed42.mp4" controls width="672"></video>

	\| Steps \| CFG \| Size \| Frames \| Seed \| Sampler \| Shift (video / audio) \|
	\|---\|---\|---\|---\|---\|---\|---\|
	\| 8 \| 1.0 \| 1344×768 \| 124 at 24 fps \| 42 \| DMAD \| 12 / 2 \|

	Same prompt and render setup as the 4-step FL2VA example.

	### Ref2VA, 8 steps

	<video src="https://huggingface.co/Spectrallic/MiniMax-H3-DMAD-int8-convrot/resolve/main/samples/stormy_boat_8step_1344x768_seed42.mp4" controls width="672"></video>

	\| Steps \| CFG \| Size \| Frames \| Seed \| Sampler \| Shift (video / audio) \|
	\|---\|---\|---\|---\|---\|---\|---\|
	\| 8 \| 1.0 \| 1344×768 \| 124 at 24 fps \| 42 \| DMAD \| 12 / 2 \|

	Same prompt and render setup as the 4-step Ref2VA example. `<Picture 1>` is the first frame of the 8-step FL2VA example above.

	## Sampling

	Use each DMAD student's own operating point: 4 steps for the 4-step files and 8 steps for the 8-step files,
	CFG 1 (no classifier-free guidance), video time shift 12 and audio shift 2. Workflows are in the [DMAD repository](https://huggingface.co/ZhengmingYu/DMAD).

	Both examples were rendered in Nox with the DMAD sampler. For Ref2VA, refer to references in the prompt as `<Picture i>`,
	`<Video k>` and `<Audio j>`.

	## How they were made

	`<variant>` below is `fl2va` or `ref2va`, and `<steps>` is `4step` or `8step`.

	1. Source:
	- Base: `diffusion_models/minimax_h3_<variant>_pruned_bf16.safetensors` from Comfy-Org/MiniMax-H3 (Ref2VA: revision
	`912b6e38`).
	- LoRA: `minimax_h3/dmad_minimax_h3_<steps>_full_critic_comfyui.safetensors` from ZhengmingYu/DMAD (revision
	`b08c2bf3`). These are the authors' exact conversions of `full_critic` and `8step_full_critic`, with no rank
	reduction: rank 128, with q/k/v fused into `attn.qkv_proj` adapters of rank 384.
	- Layout reference: `diffusion_models/minimax_h3_<variant>_pruned_int8_convrot.safetensors` from Comfy-Org/MiniMax-H3
	(FL2VA: revision `d6cfb4e5`; Ref2VA: revision `912b6e38`).
	2. Merge: the LoRA is merged at strength 1.0 into its 208 targets: `attn.{qkv_proj,out_proj}` and `mlp.{fc1,fc2}`
	in all 50 blocks and both `token_refiner` blocks. Each merge computes `W' = W + (alpha/rank) · up @ down` in FP32.
	3. Quantize: the 200 block linears are rotated with a group-256 Hadamard ConvRot, then quantized to INT8 with
	per-row scales, directly from the FP32 merge result. The 8 merged `token_refiner` weights go back to BF16, because
	that layer stays unquantized in the Comfy-Org file.
	4. Copy everything else: tensors the LoRA doesn't touch are copied bit for bit from the layout reference.

	### Checks

	- Against the Comfy-Org INT8 file, byte for byte (all four files): all 932 tensors match in name, order, dtype, shape and offset. Only
	408 tensors have different bytes: the 200 INT8 weights, their 200 scales and the 8 `token_refiner` weights. The other
	524 are identical, including all 200 `.comfy_quant` tensors.
	- LoRA presence: this projects the change from the base onto the LoRA delta, where 1 means fully
	present. 4 steps: the overall result is 0.9977. Of the 208 layers, 205 are within 1 ± 0.1. The other three are
	`token_refiner.blocks.0.mlp.fc2` (0.647), `token_refiner.blocks.1.attn.qkv_proj` (0.896) and
	`token_refiner.blocks.1.attn.out_proj` (0.899).
	8 steps: the overall result is 0.9912 for FL2VA and 0.9913 for Ref2VA. All 200 block layers are within 1 ± 0.1. The 8
	`token_refiner` layers fall between 0.36 and 0.73: their 8-step delta is only about 0.07% of the weight, which is below
	BF16 rounding, so part of it rounds away when they are stored in their Comfy-Org dtype.

	## Credits & license

	- Base model: [MiniMaxAI/MiniMax-H3](https://huggingface.co/MiniMaxAI/MiniMax-H3), repackaged by
	[Comfy-Org/MiniMax-H3](https://huggingface.co/Comfy-Org/MiniMax-H3)
	- Distillation: [ZhengmingYu/DMAD](https://huggingface.co/ZhengmingYu/DMAD)
	([paper](https://arxiv.org/abs/2610.02188), [code](https://github.com/Yzmblog/DMAD))

	This derivative is released under the [MiniMax H3 Community License Agreement](LICENSE), which includes territorial
	restrictions. See `LICENSE` and `NOTICE`.

	Powered by MiniMax H3.