Add full-encoder MXFP8 RTN FLUX.2 model and tested inference examples

54daaf9 verified 3 days ago

5.98 kB

---
base_model: black-forest-labs/FLUX.2-dev
pipeline_tag: text-to-image
library_name: diffusers
license: other
license_name: flux-2-dev-non-commercial-license
license_link: https://huggingface.co/black-forest-labs/FLUX.2-dev/blob/main/LICENSE.md
tags:
- auto-round
- mxfp8
- rtn
- multimodal
---
# FLUX.2-dev-MXFP8-RTN-FullEncoder-AutoRound

## Model Details

This is an **MXFP8 W8A8** quantization of
[black-forest-labs/FLUX.2-dev](https://huggingface.co/black-forest-labs/FLUX.2-dev),
generated with [AutoRound](https://github.com/intel/auto-round) using zero-shot
RTN. Please follow the license of the original model.

Unlike the transformer-only `FLUX.2-dev-MXFP8-RTN-AutoRound`, this variant
quantizes both the diffusion transformer blocks and the full multimodal encoder.

- **Quantized:** diffusion transformer blocks (192 Linear layers), encoder
  language layers (280), vision tower layers (168), and multimodal projector (3).
- **Kept in BF16:** VAE, embeddings, norms, encoder lm_head, vision patch
  convolution, and non-block transformer projections, including modulation/output.
- **Settings:** group size 32, dynamic activation quantization, `iters=0`,
  `model_free=True`, `disable_opt_rtn=True`. No calibration or smoothing.
- **AutoRound commit:** `8869842f9741538ea729849d5c23592bdb605e61`.

## How to Use

Run the following commands from the parent of the downloaded model directory.
Name the downloaded model directory `FLUX.2-dev-MXFP8-RTN-FullEncoder-AutoRound`. The bundled
scripts handle component quantization configuration and checkpoint index layout.

### Text-to-Image — vLLM-Omni

Evaluated environment: vLLM 0.32.0, vLLM-Omni 0.32.0, Transformers 5.19,
Diffusers 0.40, PyTorch 2.13+cu130 on NVIDIA B200.

```bash
CUDA_VISIBLE_DEVICES=0 python FLUX.2-dev-MXFP8-RTN-FullEncoder-AutoRound/examples/text_to_image.py \
    --model FLUX.2-dev-MXFP8-RTN-FullEncoder-AutoRound \
    --prompt "A red colored car." \
    --output output.png
```

Defaults: 1024 × 1024, 50 steps, guidance 4.0, seed 42. The script sets
`CUDA_LAUNCH_BLOCKING=1`, uses component-specific configs, and hides stale
duplicate index files in a temporary runtime view.

### Image Question Answering — Transformers + AutoRound

Evaluated environment: Transformers 5.14.1, PyTorch 2.13 and the AutoRound commit
above, with the `auto_round:torch_mxfp8` backend.

```bash
CUDA_VISIBLE_DEVICES=0 python FLUX.2-dev-MXFP8-RTN-FullEncoder-AutoRound/examples/scienceqa.py \
    --model FLUX.2-dev-MXFP8-RTN-FullEncoder-AutoRound \
    --index 0
```

Loads one question from
[ScienceQA-IMG-Test-2017](https://huggingface.co/datasets/changwangss/ScienceQA-IMG-Test-2017)
and selects an answer using next-token logits restricted to the valid option
letters. This is forced-choice inference, not free-form generation.

**Backend note:** ScienceQA was evaluated with Transformers, not vLLM-Omni.
The verified Omni T2I encoder skips the vision tower and projector; a full
vLLM-Omni multimodal question-answering recipe has not been validated here.
The Transformers script explicitly enables all 451 quantized encoder Linear
layers, including the projector omitted by the serialized block list.

## Evaluation

### Text-to-Image — DrawBench (200 Images)

| Model | CLIP | CLIP-IQA | ImageReward |
|---|---:|---:|---:|
| BF16 | 32.5236 | 71.7570 | 1.1716 |
| Upstream transformer-only MXFP8 | 32.4859 | 72.0392 | 1.1779 |
| Transformer + language-only encoder MXFP8 | 32.4436 | 71.9781 | 1.1547 |
| **Full encoder MXFP8 (this model)** | **32.4436** | **71.9781** | **1.1547** |

Same prompts, 1024 × 1024, 50 steps, guidance 4.0, seed 42 + image ID.
Metrics follow the neural-compressor PR2625 example: CLIP ViT-B/32 image-text
logits, three-pair CLIP-IQA (×100), and ImageReward-v1.0.

Compared with BF16, CLIP decreases by 0.25%, CLIP-IQA increases by 0.31%, and
ImageReward decreases by 1.44%. The full/language-only encoder scores match at
four decimal places; pixel-level equality was not checked. Quantizing the visual
modules does not accelerate the current Omni T2I path, which does not use them.

### Multimodal Encoder — ScienceQA (2,017 Image Questions)

| Encoder | Correct / Total | Accuracy |
|---|---:|---:|
| BF16 | 1,841 / 2,017 | 91.2742% |
| **Full encoder MXFP8** | **1,842 / 2,017** | **91.3237%** |

The full image-only test split was evaluated with identical image, question,
options and hint inputs. Answers, lectures and solutions were not model inputs.
All 171 visual/projector Linear layers executed for every question with finite
outputs, and all 451 quantized encoder layers loaded without key mismatches.

Relative accuracy loss is **−0.0543%**, meeting the **loss < 1%** criterion on
this task. One additional correct answer is not a significant improvement or
evidence of universal losslessness. ImageReward does not meet a per-metric 1%
loss threshold. An earlier 200-question subset gave 92.5% BF16 versus 90.5% MXFP8;
the complete split is the primary result.

Data: [changwangss/ScienceQA-IMG-Test-2017](https://huggingface.co/datasets/changwangss/ScienceQA-IMG-Test-2017),
revision `d21962594f7ae2cd2819fe48e95903d8a3aef5a5`.
Source: `lmms-lab-encoder/ScienceQA`, ScienceQA-IMG/test,
revision `69dd4d6b67373d38f96a5badd5d24d0eb5bcdc50`. No sampling or filtering.

## Publication Notes

All pipeline assets are included as regular files, together with the two tested
example scripts. No external filesystem symlinks are required.

## Limitations and License

Results apply to the stated backends and evaluation protocols. Free-form VQA,
image editing and other backends have not been validated. Generated content may
be inaccurate; downstream applications require appropriate quality and safety testing.

Follow the [original FLUX.2-dev license](https://huggingface.co/black-forest-labs/FLUX.2-dev/blob/main/LICENSE.md).
Quantization grants no additional rights. ScienceQA is separately distributed
under [CC BY-NC-SA 4.0](https://scienceqa.github.io/).

	---
	base_model: black-forest-labs/FLUX.2-dev
	pipeline_tag: text-to-image
	library_name: diffusers
	license: other
	license_name: flux-2-dev-non-commercial-license
	license_link: https://huggingface.co/black-forest-labs/FLUX.2-dev/blob/main/LICENSE.md
	tags:
	- auto-round
	- mxfp8
	- rtn
	- multimodal
	---

	# FLUX.2-dev-MXFP8-RTN-FullEncoder-AutoRound

	## Model Details

	This is an MXFP8 W8A8 quantization of
	[black-forest-labs/FLUX.2-dev](https://huggingface.co/black-forest-labs/FLUX.2-dev),
	generated with [AutoRound](https://github.com/intel/auto-round) using zero-shot
	RTN. Please follow the license of the original model.

	Unlike the transformer-only `FLUX.2-dev-MXFP8-RTN-AutoRound`, this variant
	quantizes both the diffusion transformer blocks and the full multimodal encoder.

	- Quantized: diffusion transformer blocks (192 Linear layers), encoder
	language layers (280), vision tower layers (168), and multimodal projector (3).
	- Kept in BF16: VAE, embeddings, norms, encoder lm_head, vision patch
	convolution, and non-block transformer projections, including modulation/output.
	- Settings: group size 32, dynamic activation quantization, `iters=0`,
	`model_free=True`, `disable_opt_rtn=True`. No calibration or smoothing.
	- AutoRound commit: `8869842f9741538ea729849d5c23592bdb605e61`.

	## How to Use

	Run the following commands from the parent of the downloaded model directory.
	Name the downloaded model directory `FLUX.2-dev-MXFP8-RTN-FullEncoder-AutoRound`. The bundled
	scripts handle component quantization configuration and checkpoint index layout.

	### Text-to-Image — vLLM-Omni

	Evaluated environment: vLLM 0.32.0, vLLM-Omni 0.32.0, Transformers 5.19,
	Diffusers 0.40, PyTorch 2.13+cu130 on NVIDIA B200.

	```bash
	CUDA_VISIBLE_DEVICES=0 python FLUX.2-dev-MXFP8-RTN-FullEncoder-AutoRound/examples/text_to_image.py \
	--model FLUX.2-dev-MXFP8-RTN-FullEncoder-AutoRound \
	--prompt "A red colored car." \
	--output output.png
	```

	Defaults: 1024 × 1024, 50 steps, guidance 4.0, seed 42. The script sets
	`CUDA_LAUNCH_BLOCKING=1`, uses component-specific configs, and hides stale
	duplicate index files in a temporary runtime view.

	### Image Question Answering — Transformers + AutoRound

	Evaluated environment: Transformers 5.14.1, PyTorch 2.13 and the AutoRound commit
	above, with the `auto_round:torch_mxfp8` backend.

	```bash
	CUDA_VISIBLE_DEVICES=0 python FLUX.2-dev-MXFP8-RTN-FullEncoder-AutoRound/examples/scienceqa.py \
	--model FLUX.2-dev-MXFP8-RTN-FullEncoder-AutoRound \
	--index 0
	```

	Loads one question from
	[ScienceQA-IMG-Test-2017](https://huggingface.co/datasets/changwangss/ScienceQA-IMG-Test-2017)
	and selects an answer using next-token logits restricted to the valid option
	letters. This is forced-choice inference, not free-form generation.

	Backend note: ScienceQA was evaluated with Transformers, not vLLM-Omni.
	The verified Omni T2I encoder skips the vision tower and projector; a full
	vLLM-Omni multimodal question-answering recipe has not been validated here.
	The Transformers script explicitly enables all 451 quantized encoder Linear
	layers, including the projector omitted by the serialized block list.

	## Evaluation

	### Text-to-Image — DrawBench (200 Images)

	\| Model \| CLIP \| CLIP-IQA \| ImageReward \|
	\|---\|---:\|---:\|---:\|
	\| BF16 \| 32.5236 \| 71.7570 \| 1.1716 \|
	\| Upstream transformer-only MXFP8 \| 32.4859 \| 72.0392 \| 1.1779 \|
	\| Transformer + language-only encoder MXFP8 \| 32.4436 \| 71.9781 \| 1.1547 \|
	\| Full encoder MXFP8 (this model) \| 32.4436 \| 71.9781 \| 1.1547 \|

	Same prompts, 1024 × 1024, 50 steps, guidance 4.0, seed 42 + image ID.
	Metrics follow the neural-compressor PR2625 example: CLIP ViT-B/32 image-text
	logits, three-pair CLIP-IQA (×100), and ImageReward-v1.0.

	Compared with BF16, CLIP decreases by 0.25%, CLIP-IQA increases by 0.31%, and
	ImageReward decreases by 1.44%. The full/language-only encoder scores match at
	four decimal places; pixel-level equality was not checked. Quantizing the visual
	modules does not accelerate the current Omni T2I path, which does not use them.

	### Multimodal Encoder — ScienceQA (2,017 Image Questions)

	\| Encoder \| Correct / Total \| Accuracy \|
	\|---\|---:\|---:\|
	\| BF16 \| 1,841 / 2,017 \| 91.2742% \|
	\| Full encoder MXFP8 \| 1,842 / 2,017 \| 91.3237% \|

	The full image-only test split was evaluated with identical image, question,
	options and hint inputs. Answers, lectures and solutions were not model inputs.
	All 171 visual/projector Linear layers executed for every question with finite
	outputs, and all 451 quantized encoder layers loaded without key mismatches.

	Relative accuracy loss is −0.0543%, meeting the loss < 1% criterion on
	this task. One additional correct answer is not a significant improvement or
	evidence of universal losslessness. ImageReward does not meet a per-metric 1%
	loss threshold. An earlier 200-question subset gave 92.5% BF16 versus 90.5% MXFP8;
	the complete split is the primary result.

	Data: [changwangss/ScienceQA-IMG-Test-2017](https://huggingface.co/datasets/changwangss/ScienceQA-IMG-Test-2017),
	revision `d21962594f7ae2cd2819fe48e95903d8a3aef5a5`.
	Source: `lmms-lab-encoder/ScienceQA`, ScienceQA-IMG/test,
	revision `69dd4d6b67373d38f96a5badd5d24d0eb5bcdc50`. No sampling or filtering.

	## Publication Notes

	All pipeline assets are included as regular files, together with the two tested
	example scripts. No external filesystem symlinks are required.

	## Limitations and License

	Results apply to the stated backends and evaluation protocols. Free-form VQA,
	image editing and other backends have not been validated. Generated content may
	be inaccurate; downstream applications require appropriate quality and safety testing.

	Follow the [original FLUX.2-dev license](https://huggingface.co/black-forest-labs/FLUX.2-dev/blob/main/LICENSE.md).
	Quantization grants no additional rights. ScienceQA is separately distributed
	under [CC BY-NC-SA 4.0](https://scienceqa.github.io/).