Blame for README.md

---
license: other
license_name: flux-non-commercial-license-v2.1
license_link: https://bfl.ai/licensing
base_model:
  - wikeeyang/Flux2-Klein-9B-True-V3
  - black-forest-labs/FLUX.2-klein-9B
base_model_relation: quantized
library_name: diffusers
pipeline_tag: text-to-image
tags:
  - flux
  - flux2
  - klein
  - fp8
  - quantized
  - optimum-quanto
language:
  - en
---

# Flux2-Klein-9B-True-V3-fp8

An FP8 build of [`wikeeyang/Flux2-Klein-9B-True-V3`](https://huggingface.co/wikeeyang/Flux2-Klein-9B-True-V3), saved in the [optimum-quanto](https://github.com/huggingface/optimum-quanto) `QuantizedDiffusersModel` layout so it loads directly into a `diffusers` `Flux2KleinPipeline` with no conversion step.

**This is a format and precision conversion only.** Nothing was retrained, distilled or merged. The weights are wikeeyang's True-V3.

> ### ⚠️ Licence — read before use
>
> These weights derive from [`black-forest-labs/FLUX.2-klein-9B`](https://huggingface.co/black-forest-labs/FLUX.2-klein-9B), released under the **FLUX Non-Commercial License v2.1**. Commercial or revenue-generating deployment requires a separate licence from Black Forest Labs — see [bfl.ai/licensing](https://bfl.ai/licensing).
>
> Only FLUX.2-klein-**4B** is Apache-2.0. The 9B models are not. The upstream True-V3 repository is tagged `apache-2.0`; that tag does not reflect the terms of the base model, and a third party cannot relicense weights they do not own. This repository is labelled to match the actual upstream terms.

## Why this exists

True-V3 ships ComfyUI single-file formats only — `bf16`, `fp8mixed`, `int8mixedrow`, GGUF, `nvfp4`, `int4_convrot`. None of them is a quanto checkpoint: `QuantizedDiffusersModel.from_pretrained()` expects quanto's own directory layout (`_data`/`_scale` tensors plus a quanto config), and quanto cannot read ComfyUI's scaled-FP8 tensor layout.

So a diffusers/quanto serving stack could not use True-V3 without converting it first. This repository is that conversion, published so nobody has to redo it.

## Usage

```python
import torch
from diffusers import Flux2KleinPipeline, Flux2Transformer2DModel
from optimum.quanto import QuantizedDiffusersModel

# quanto's Marlin FP8 kernel rejects a non-contiguous activation, which the
# Flux2 attention path produces. Apply before loading.
from optimum.quanto.nn.qlinear import QLinear
_orig = QLinear.forward
QLinear.forward = lambda self, x: _orig(self, x.contiguous())


class QuantizedFlux2KleinTransformer(QuantizedDiffusersModel):
    base_class = Flux2Transformer2DModel


pipe = Flux2KleinPipeline.from_pretrained(
    "black-forest-labs/FLUX.2-klein-9B",
    transformer=None,
    torch_dtype=torch.bfloat16,
).to("cuda")

pipe.transformer = QuantizedFlux2KleinTransformer.from_pretrained(
    "ModelsLab/Flux2-Klein-9B-True-V3-fp8"
).to("cuda")

image = pipe(
    prompt="a vintage storefront window with a hand-painted gold sign that reads "
           '"KLEIN COFFEE ROASTERS EST 1974", rainy street reflection',
    height=1024, width=1024,
    num_inference_steps=4,
    generator=torch.Generator("cuda").manual_seed(1234),
).images[0]
image.save("out.png")
```

### Tighter VRAM

The transformer is 9.08 GB. The bf16 text encoder is another ~16 GB, which will not fit alongside it on a 24 GB card. Quantise it the same way:

```python
from optimum.quanto import QuantizedModelForCausalLM, qfloat8
from transformers import AutoModelForCausalLM

te = AutoModelForCausalLM.from_pretrained(
    "black-forest-labs/FLUX.2-klein-9B", subfolder="text_encoder",
    torch_dtype=torch.bfloat16)
pipe.text_encoder = QuantizedModelForCausalLM.quantize(
    te, weights=qfloat8).to("cuda")
```

FP8 transformer + FP8 text encoder is ~18 GB resident. Measured peak allocation on an RTX 3090 at 1024×1024, 4 steps, with both resident: **20.66 GB** — so it fits a 24 GB card, with roughly 3 GB spare.

## Measured against the stock FP8 checkpoint

![stock klein-9B FP8 vs True-V3 FP8, same prompt and seed](comparison.jpg)

*Same prompt, same seed, same 4 steps, same quantiser. Top row is the stock FP8 checkpoint, bottom row is this one. Note the misspelled "COFFFEE" in the stock render.*

Both checkpoints were run through an identical harness: same architecture, same quantiser, same 4 steps, same guidance, same seeds, and prompt embeddings encoded once and cached so both saw bit-identical conditioning. 12 prompts × 3 seeds = **36 timed renders each**, 1024×1024, on an RTX 3090.

| | stock klein-9B FP8 | True-V3 FP8 | Δ |
|---|---|---|---|
| Aesthetic (LAION v2) | 5.996 | **6.167** | +0.171 |
| CLIP adherence (×100) | 28.98 | **29.65** | +0.67 |
| Reference retention¹ | 56.6 | **60.6** | +4.0 |
| Median latency / 1024² | 6.29 s | 6.31 s | +0.2% |
| On-disk size | 9.08 GB | 9.08 GB | — |

¹ CLIP image–image cosine ×100 between each edit and its reference images, over a two-reference editing run (four scenes × two seeds), scored on the render set shared by every variant tested.

The latency difference is inside run-to-run noise — across eight variants benchmarked the same way, the whole spread was 0.06 s. Same architecture, same step count, same FP8 kernels, so this is a quality change at fixed cost rather than a trade.

Qualitatively, the clearest wins over stock are **rendered text** (the sign prompt above is misspelled "COFFFEE" by the stock checkpoint and correct here), **hands**, and **product/material detail**. Colour is slightly less saturated and more filmic.

**Caveat on the metrics:** the LAION aesthetic predictor rewards contrast and saturation, so it is not a neutral judge of photographic realism — a crunchier model can score well while rendering worse text. Treat the table as a ranking aid and look at your own prompts.

## Verification

This checkpoint was checked by regenerating four benchmark prompts at a fixed seed and diffing them
against the renders produced by the original bf16 True-V3 weights in the earlier benchmark run:

| prompt | max abs diff | mean abs diff | PSNR |
|---|---|---|---|
| text | 0 | 0.0000 | ∞ |
| portrait | 0 | 0.0000 | ∞ |
| hands | 0 | 0.0000 | ∞ |
| product | 0 | 0.0000 | ∞ |

**Bit-identical** on all four. The conversion changes the container, not the output.

## How this was built

Reproducible in about a minute of compute once the 18 GB source file is local (63 s measured: load, quantise, save):

```python
import torch
from diffusers import Flux2Transformer2DModel
from optimum.quanto import QuantizedDiffusersModel, qfloat8
from huggingface_hub import hf_hub_download


class QuantizedFlux2KleinTransformer(QuantizedDiffusersModel):
    base_class = Flux2Transformer2DModel


path = hf_hub_download("wikeeyang/Flux2-Klein-9B-True-V3",
                       "Flux2-Klein-9B-True-V3-bf16.safetensors")

tr = Flux2Transformer2DModel.from_single_file(
    path,
    config="black-forest-labs/FLUX.2-klein-9B", subfolder="transformer",
    torch_dtype=torch.bfloat16,
)
QuantizedFlux2KleinTransformer.quantize(tr, weights=qfloat8).save_pretrained("out")
```

## Notes and caveats

- **Ampere (RTX 3090 / A100):** these cards have no native FP8 tensor cores. quanto routes through its Marlin FP8 GEMM, which does run on `sm_80`/`sm_86` — so FP8 here buys memory, and the speed is comparable to the equivalent bf16 path rather than dramatically faster. Ada and Hopper get more from it.
- **The contiguity patch in the usage snippet is required**, not optional. Without it the Marlin kernel raises `RuntimeError: A is not contiguous` on the first forward pass.
- **LoRAs must be fused before quantisation.** Calling `load_lora_weights()` on an already-quantised transformer does not work. Load the bf16 transformer, `fuse_lora()`, then quantise.
- **Only the transformer is replaced.** Text encoder, VAE, scheduler and tokenizer come from the base `FLUX.2-klein-9B` repo unchanged.
- **`guidance_scale` is ignored.** Klein is step-distilled, and `diffusers` emits *"Guidance scale 4.0 is ignored for step-wise distilled models"* and drops the value. Passing it does nothing; do not expect it to behave as a tuning knob.

## Credits

- [**wikeeyang**](https://huggingface.co/wikeeyang) — the True-V3 finetune. All of the quality improvement is theirs; this repository only changes the container.
- [**Black Forest Labs**](https://huggingface.co/black-forest-labs) — FLUX.2-klein-9B.
- [**optimum-quanto**](https://github.com/huggingface/optimum-quanto) — the quantiser and the checkpoint format.

Converted and published by [ModelsLab](https://huggingface.co/ModelsLab).