FP8 (optimum-quanto) conversion of wikeeyang/Flux2-Klein-9B-True-V3

b37a2f5 verified 2 months ago

8.58 kB

---
license: other
license_name: flux-non-commercial-license-v2.1
license_link: https://bfl.ai/licensing
base_model:
  - wikeeyang/Flux2-Klein-9B-True-V3
  - black-forest-labs/FLUX.2-klein-9B
base_model_relation: quantized
library_name: diffusers
pipeline_tag: text-to-image
tags:
  - flux
  - flux2
  - klein
  - fp8
  - quantized
  - optimum-quanto
language:
  - en
---
# Flux2-Klein-9B-True-V3-fp8

An FP8 build of [`wikeeyang/Flux2-Klein-9B-True-V3`](https://huggingface.co/wikeeyang/Flux2-Klein-9B-True-V3), saved in the [optimum-quanto](https://github.com/huggingface/optimum-quanto) `QuantizedDiffusersModel` layout so it loads directly into a `diffusers` `Flux2KleinPipeline` with no conversion step.

**This is a format and precision conversion only.** Nothing was retrained, distilled or merged. The weights are wikeeyang's True-V3.

> ### ⚠️ Licence — read before use
>
> These weights derive from [`black-forest-labs/FLUX.2-klein-9B`](https://huggingface.co/black-forest-labs/FLUX.2-klein-9B), released under the **FLUX Non-Commercial License v2.1**. Commercial or revenue-generating deployment requires a separate licence from Black Forest Labs — see [bfl.ai/licensing](https://bfl.ai/licensing).
>
> Only FLUX.2-klein-**4B** is Apache-2.0. The 9B models are not. The upstream True-V3 repository is tagged `apache-2.0`; that tag does not reflect the terms of the base model, and a third party cannot relicense weights they do not own. This repository is labelled to match the actual upstream terms.

## Why this exists

True-V3 ships ComfyUI single-file formats only — `bf16`, `fp8mixed`, `int8mixedrow`, GGUF, `nvfp4`, `int4_convrot`. None of them is a quanto checkpoint: `QuantizedDiffusersModel.from_pretrained()` expects quanto's own directory layout (`_data`/`_scale` tensors plus a quanto config), and quanto cannot read ComfyUI's scaled-FP8 tensor layout.

So a diffusers/quanto serving stack could not use True-V3 without converting it first. This repository is that conversion, published so nobody has to redo it.

## Usage

```python
import torch
from diffusers import Flux2KleinPipeline, Flux2Transformer2DModel
from optimum.quanto import QuantizedDiffusersModel
# quanto's Marlin FP8 kernel rejects a non-contiguous activation, which the
# Flux2 attention path produces. Apply before loading.
from optimum.quanto.nn.qlinear import QLinear
_orig = QLinear.forward
QLinear.forward = lambda self, x: _orig(self, x.contiguous())
class QuantizedFlux2KleinTransformer(QuantizedDiffusersModel):
    base_class = Flux2Transformer2DModel
pipe = Flux2KleinPipeline.from_pretrained(
    "black-forest-labs/FLUX.2-klein-9B",
    transformer=None,
    torch_dtype=torch.bfloat16,
).to("cuda")
pipe.transformer = QuantizedFlux2KleinTransformer.from_pretrained(
    "ModelsLab/Flux2-Klein-9B-True-V3-fp8"
).to("cuda")
image = pipe(
    prompt="a vintage storefront window with a hand-painted gold sign that reads "
           '"KLEIN COFFEE ROASTERS EST 1974", rainy street reflection',
    height=1024, width=1024,
    num_inference_steps=4,
    generator=torch.Generator("cuda").manual_seed(1234),
).images[0]
image.save("out.png")
```

### Tighter VRAM

The transformer is 9.08 GB. The bf16 text encoder is another ~16 GB, which will not fit alongside it on a 24 GB card. Quantise it the same way:

```python
from optimum.quanto import QuantizedModelForCausalLM, qfloat8
from transformers import AutoModelForCausalLM
te = AutoModelForCausalLM.from_pretrained(
    "black-forest-labs/FLUX.2-klein-9B", subfolder="text_encoder",
    torch_dtype=torch.bfloat16)
pipe.text_encoder = QuantizedModelForCausalLM.quantize(
    te, weights=qfloat8).to("cuda")
```

FP8 transformer + FP8 text encoder is ~18 GB resident. Measured peak allocation on an RTX 3090 at 1024×1024, 4 steps, with both resident: **20.66 GB** — so it fits a 24 GB card, with roughly 3 GB spare.

## Measured against the stock FP8 checkpoint

![stock klein-9B FP8 vs True-V3 FP8, same prompt and seed](comparison.jpg)

*Same prompt, same seed, same 4 steps, same quantiser. Top row is the stock FP8 checkpoint, bottom row is this one. Note the misspelled "COFFFEE" in the stock render.*

Both checkpoints were run through an identical harness: same architecture, same quantiser, same 4 steps, same guidance, same seeds, and prompt embeddings encoded once and cached so both saw bit-identical conditioning. 12 prompts × 3 seeds = **36 timed renders each**, 1024×1024, on an RTX 3090.

| | stock klein-9B FP8 | True-V3 FP8 | Δ |
|---|---|---|---|
| Aesthetic (LAION v2) | 5.996 | **6.167** | +0.171 |
| CLIP adherence (×100) | 28.98 | **29.65** | +0.67 |
| Reference retention¹ | 56.6 | **60.6** | +4.0 |
| Median latency / 1024² | 6.29 s | 6.31 s | +0.2% |
| On-disk size | 9.08 GB | 9.08 GB | — |

¹ CLIP image–image cosine ×100 between each edit and its reference images, over a two-reference editing run (four scenes × two seeds), scored on the render set shared by every variant tested.

The latency difference is inside run-to-run noise — across eight variants benchmarked the same way, the whole spread was 0.06 s. Same architecture, same step count, same FP8 kernels, so this is a quality change at fixed cost rather than a trade.

Qualitatively, the clearest wins over stock are **rendered text** (the sign prompt above is misspelled "COFFFEE" by the stock checkpoint and correct here), **hands**, and **product/material detail**. Colour is slightly less saturated and more filmic.

**Caveat on the metrics:** the LAION aesthetic predictor rewards contrast and saturation, so it is not a neutral judge of photographic realism — a crunchier model can score well while rendering worse text. Treat the table as a ranking aid and look at your own prompts.

## Verification

This checkpoint was checked by regenerating four benchmark prompts at a fixed seed and diffing them
against the renders produced by the original bf16 True-V3 weights in the earlier benchmark run:

| prompt | max abs diff | mean abs diff | PSNR |
|---|---|---|---|
| text | 0 | 0.0000 | ∞ |
| portrait | 0 | 0.0000 | ∞ |
| hands | 0 | 0.0000 | ∞ |
| product | 0 | 0.0000 | ∞ |

**Bit-identical** on all four. The conversion changes the container, not the output.

## How this was built

Reproducible in about a minute of compute once the 18 GB source file is local (63 s measured: load, quantise, save):

```python
import torch
from diffusers import Flux2Transformer2DModel
from optimum.quanto import QuantizedDiffusersModel, qfloat8
from huggingface_hub import hf_hub_download
class QuantizedFlux2KleinTransformer(QuantizedDiffusersModel):
    base_class = Flux2Transformer2DModel
path = hf_hub_download("wikeeyang/Flux2-Klein-9B-True-V3",
                       "Flux2-Klein-9B-True-V3-bf16.safetensors")
tr = Flux2Transformer2DModel.from_single_file(
    path,
    config="black-forest-labs/FLUX.2-klein-9B", subfolder="transformer",
    torch_dtype=torch.bfloat16,
)
QuantizedFlux2KleinTransformer.quantize(tr, weights=qfloat8).save_pretrained("out")
```

## Notes and caveats

- **Ampere (RTX 3090 / A100):** these cards have no native FP8 tensor cores. quanto routes through its Marlin FP8 GEMM, which does run on `sm_80`/`sm_86` — so FP8 here buys memory, and the speed is comparable to the equivalent bf16 path rather than dramatically faster. Ada and Hopper get more from it.
- **The contiguity patch in the usage snippet is required**, not optional. Without it the Marlin kernel raises `RuntimeError: A is not contiguous` on the first forward pass.
- **LoRAs must be fused before quantisation.** Calling `load_lora_weights()` on an already-quantised transformer does not work. Load the bf16 transformer, `fuse_lora()`, then quantise.
- **Only the transformer is replaced.** Text encoder, VAE, scheduler and tokenizer come from the base `FLUX.2-klein-9B` repo unchanged.
- **`guidance_scale` is ignored.** Klein is step-distilled, and `diffusers` emits *"Guidance scale 4.0 is ignored for step-wise distilled models"* and drops the value. Passing it does nothing; do not expect it to behave as a tuning knob.
## Credits
- [**wikeeyang**](https://huggingface.co/wikeeyang) — the True-V3 finetune. All of the quality improvement is theirs; this repository only changes the container.
- [**Black Forest Labs**](https://huggingface.co/black-forest-labs) — FLUX.2-klein-9B.
- [**optimum-quanto**](https://github.com/huggingface/optimum-quanto) — the quantiser and the checkpoint format.
Converted and published by [ModelsLab](https://huggingface.co/ModelsLab).

	---
	license: other
	license_name: flux-non-commercial-license-v2.1
	license_link: https://bfl.ai/licensing
	base_model:
	- wikeeyang/Flux2-Klein-9B-True-V3
	- black-forest-labs/FLUX.2-klein-9B
	base_model_relation: quantized
	library_name: diffusers
	pipeline_tag: text-to-image
	tags:
	- flux
	- flux2
	- klein
	- fp8
	- quantized
	- optimum-quanto
	language:
	- en
	---

	# Flux2-Klein-9B-True-V3-fp8

	An FP8 build of [`wikeeyang/Flux2-Klein-9B-True-V3`](https://huggingface.co/wikeeyang/Flux2-Klein-9B-True-V3), saved in the [optimum-quanto](https://github.com/huggingface/optimum-quanto) `QuantizedDiffusersModel` layout so it loads directly into a `diffusers` `Flux2KleinPipeline` with no conversion step.

	This is a format and precision conversion only. Nothing was retrained, distilled or merged. The weights are wikeeyang's True-V3.

	> ### ⚠️ Licence — read before use
	>
	> These weights derive from [`black-forest-labs/FLUX.2-klein-9B`](https://huggingface.co/black-forest-labs/FLUX.2-klein-9B), released under the FLUX Non-Commercial License v2.1. Commercial or revenue-generating deployment requires a separate licence from Black Forest Labs — see [bfl.ai/licensing](https://bfl.ai/licensing).
	>
	> Only FLUX.2-klein-4B is Apache-2.0. The 9B models are not. The upstream True-V3 repository is tagged `apache-2.0`; that tag does not reflect the terms of the base model, and a third party cannot relicense weights they do not own. This repository is labelled to match the actual upstream terms.

	## Why this exists

	True-V3 ships ComfyUI single-file formats only — `bf16`, `fp8mixed`, `int8mixedrow`, GGUF, `nvfp4`, `int4_convrot`. None of them is a quanto checkpoint: `QuantizedDiffusersModel.from_pretrained()` expects quanto's own directory layout (`_data`/`_scale` tensors plus a quanto config), and quanto cannot read ComfyUI's scaled-FP8 tensor layout.

	So a diffusers/quanto serving stack could not use True-V3 without converting it first. This repository is that conversion, published so nobody has to redo it.

	## Usage

	```python
	import torch
	from diffusers import Flux2KleinPipeline, Flux2Transformer2DModel
	from optimum.quanto import QuantizedDiffusersModel

	# quanto's Marlin FP8 kernel rejects a non-contiguous activation, which the
	# Flux2 attention path produces. Apply before loading.
	from optimum.quanto.nn.qlinear import QLinear
	_orig = QLinear.forward
	QLinear.forward = lambda self, x: _orig(self, x.contiguous())


	class QuantizedFlux2KleinTransformer(QuantizedDiffusersModel):
	base_class = Flux2Transformer2DModel


	pipe = Flux2KleinPipeline.from_pretrained(
	"black-forest-labs/FLUX.2-klein-9B",
	transformer=None,
	torch_dtype=torch.bfloat16,
	).to("cuda")

	pipe.transformer = QuantizedFlux2KleinTransformer.from_pretrained(
	"ModelsLab/Flux2-Klein-9B-True-V3-fp8"
	).to("cuda")

	image = pipe(
	prompt="a vintage storefront window with a hand-painted gold sign that reads "
	'"KLEIN COFFEE ROASTERS EST 1974", rainy street reflection',
	height=1024, width=1024,
	num_inference_steps=4,
	generator=torch.Generator("cuda").manual_seed(1234),
	).images[0]
	image.save("out.png")
	```

	### Tighter VRAM

	The transformer is 9.08 GB. The bf16 text encoder is another ~16 GB, which will not fit alongside it on a 24 GB card. Quantise it the same way:

	```python
	from optimum.quanto import QuantizedModelForCausalLM, qfloat8
	from transformers import AutoModelForCausalLM

	te = AutoModelForCausalLM.from_pretrained(
	"black-forest-labs/FLUX.2-klein-9B", subfolder="text_encoder",
	torch_dtype=torch.bfloat16)
	pipe.text_encoder = QuantizedModelForCausalLM.quantize(
	te, weights=qfloat8).to("cuda")
	```

	FP8 transformer + FP8 text encoder is ~18 GB resident. Measured peak allocation on an RTX 3090 at 1024×1024, 4 steps, with both resident: 20.66 GB — so it fits a 24 GB card, with roughly 3 GB spare.

	## Measured against the stock FP8 checkpoint

	![stock klein-9B FP8 vs True-V3 FP8, same prompt and seed](comparison.jpg)

	Same prompt, same seed, same 4 steps, same quantiser. Top row is the stock FP8 checkpoint, bottom row is this one. Note the misspelled "COFFFEE" in the stock render.

	Both checkpoints were run through an identical harness: same architecture, same quantiser, same 4 steps, same guidance, same seeds, and prompt embeddings encoded once and cached so both saw bit-identical conditioning. 12 prompts × 3 seeds = 36 timed renders each, 1024×1024, on an RTX 3090.

	\| \| stock klein-9B FP8 \| True-V3 FP8 \| Δ \|
	\|---\|---\|---\|---\|
	\| Aesthetic (LAION v2) \| 5.996 \| 6.167 \| +0.171 \|
	\| CLIP adherence (×100) \| 28.98 \| 29.65 \| +0.67 \|
	\| Reference retention¹ \| 56.6 \| 60.6 \| +4.0 \|
	\| Median latency / 1024² \| 6.29 s \| 6.31 s \| +0.2% \|
	\| On-disk size \| 9.08 GB \| 9.08 GB \| — \|

	¹ CLIP image–image cosine ×100 between each edit and its reference images, over a two-reference editing run (four scenes × two seeds), scored on the render set shared by every variant tested.

	The latency difference is inside run-to-run noise — across eight variants benchmarked the same way, the whole spread was 0.06 s. Same architecture, same step count, same FP8 kernels, so this is a quality change at fixed cost rather than a trade.

	Qualitatively, the clearest wins over stock are rendered text (the sign prompt above is misspelled "COFFFEE" by the stock checkpoint and correct here), hands, and product/material detail. Colour is slightly less saturated and more filmic.

	Caveat on the metrics: the LAION aesthetic predictor rewards contrast and saturation, so it is not a neutral judge of photographic realism — a crunchier model can score well while rendering worse text. Treat the table as a ranking aid and look at your own prompts.

	## Verification

	This checkpoint was checked by regenerating four benchmark prompts at a fixed seed and diffing them
	against the renders produced by the original bf16 True-V3 weights in the earlier benchmark run:

	\| prompt \| max abs diff \| mean abs diff \| PSNR \|
	\|---\|---\|---\|---\|
	\| text \| 0 \| 0.0000 \| ∞ \|
	\| portrait \| 0 \| 0.0000 \| ∞ \|
	\| hands \| 0 \| 0.0000 \| ∞ \|
	\| product \| 0 \| 0.0000 \| ∞ \|

	Bit-identical on all four. The conversion changes the container, not the output.

	## How this was built

	Reproducible in about a minute of compute once the 18 GB source file is local (63 s measured: load, quantise, save):

	```python
	import torch
	from diffusers import Flux2Transformer2DModel
	from optimum.quanto import QuantizedDiffusersModel, qfloat8
	from huggingface_hub import hf_hub_download


	class QuantizedFlux2KleinTransformer(QuantizedDiffusersModel):
	base_class = Flux2Transformer2DModel


	path = hf_hub_download("wikeeyang/Flux2-Klein-9B-True-V3",
	"Flux2-Klein-9B-True-V3-bf16.safetensors")

	tr = Flux2Transformer2DModel.from_single_file(
	path,
	config="black-forest-labs/FLUX.2-klein-9B", subfolder="transformer",
	torch_dtype=torch.bfloat16,
	)
	QuantizedFlux2KleinTransformer.quantize(tr, weights=qfloat8).save_pretrained("out")
	```

	## Notes and caveats

	- Ampere (RTX 3090 / A100): these cards have no native FP8 tensor cores. quanto routes through its Marlin FP8 GEMM, which does run on `sm_80`/`sm_86` — so FP8 here buys memory, and the speed is comparable to the equivalent bf16 path rather than dramatically faster. Ada and Hopper get more from it.
	- The contiguity patch in the usage snippet is required, not optional. Without it the Marlin kernel raises `RuntimeError: A is not contiguous` on the first forward pass.
	- LoRAs must be fused before quantisation. Calling `load_lora_weights()` on an already-quantised transformer does not work. Load the bf16 transformer, `fuse_lora()`, then quantise.
	- Only the transformer is replaced. Text encoder, VAE, scheduler and tokenizer come from the base `FLUX.2-klein-9B` repo unchanged.
	- `guidance_scale` is ignored. Klein is step-distilled, and `diffusers` emits "Guidance scale 4.0 is ignored for step-wise distilled models" and drops the value. Passing it does nothing; do not expect it to behave as a tuning knob.

	## Credits

	- [wikeeyang](https://huggingface.co/wikeeyang) — the True-V3 finetune. All of the quality improvement is theirs; this repository only changes the container.
	- [Black Forest Labs](https://huggingface.co/black-forest-labs) — FLUX.2-klein-9B.
	- [optimum-quanto](https://github.com/huggingface/optimum-quanto) — the quantiser and the checkpoint format.

	Converted and published by [ModelsLab](https://huggingface.co/ModelsLab).