mlx-flux2-klein-4b-aio-4bit / conversion-guide.md
Devserhii's picture
Document Image to Image reference editing and verify runtime commit 646b518
cde0259 verified
|
Raw History Blame Contribute Delete
21 kB
# FLUX.2 Klein 4B AIO: conversion and LocalMuse integration
## Published model
| Field | Value |
| --- | --- |
| App name / ID | **FLUX.2 Klein 4B** / `flux2_klein_4b_aio_mlx` |
| Source | [Civitai model 2327389, version 2618128](https://civitai.com/models/2327389?modelVersionId=2618128), BF16 file 2511488 |
| Source version | `Flux.2-4B-Distilled-AIO`, repack by SeeSeeLP / SeeSee21 |
| HF repository | [LocalMuseAI/mlx-flux2-klein-4b-aio-4bit](https://huggingface.co/LocalMuseAI/mlx-flux2-klein-4b-aio-4bit) |
| Immutable runtime commit | `646b51828f39b2541fbfcc643d6984d199c6f6a5` (first six files unchanged since `c2abb5051572bcda02841b3e24af3e3fe9911d57`) |
| Runtime / backend | `flux2Klein4B4Bit` / `flux2KleinMLX` |
| Payload | **4,560,019,137 bytes** (4.56 GB / 4.25 GiB), seven authenticated files |
| Defaults | **1024 × 1024, 4 steps, CFG 1, Euler / Simple** |
| Allowed steps | 4–6 |
| Workflows | Text to Image; Image to Image as reference editing. One image per request |
| Device floor | 12 GiB RAM; physical iPhone validation pending |
This is an AIO repack of the official step-distilled **Klein 4B**, not the 9B
model or the undistilled base. The author's recommendations are 4–6 steps,
CFG 1, Euler, Simple or normal scheduling, and native 1024 square. LocalMuse
implements the Simple variant. Existing Bonsai, Z-Image, SD and SDXL catalog
entries retain their previous IDs, manifests, resolutions and defaults.
The model, Qwen3 text encoder and official VAE are Apache-2.0. The source card
permits redistribution and derivatives; exact metadata is in `source.json`.
Preserve `LICENSE`, `NOTICE`, the author attribution and the BFL/Qwen notices
when redistributing converted weights. Runtime code adapted from Duration AI
retains its Apache license and complete upstream NOTICE. ComfyUI is only an
external development reference; no ComfyUI implementation is included in the app.
## What is converted
- DiT: 3072 hidden dimensions, 24 heads, five double-stream and twenty
single-stream blocks. Attention and FFN matrices use ordinary affine INT4,
group 64; sensitive input/output, timestep and modulation layers remain BF16.
- Qwen3-4B: INT4/group64 including row-wise embedding lookup. Raw hidden states
after layers **9, 18 and 27** are concatenated to 7680 channels. Layers after
the last tap, the final language normalization and language head are unused
and omitted. No entire embedding-table dequantization occurs at runtime.
- Tokenizer: official BFL-pinned Qwen3 files, empty thinking chat template,
right padding to 512 tokens, causal and padding attention masks.
- VAE: original **32-channel** AIO decoder, encoder and packed 128-channel
latent BN statistics retained in **FP32**; see the projection note below.
Convolution weights change layout to MLX OHWI; attention weights become 2D.
The encoder is a separate `vae_encoder.safetensors` (its nested `quant_conv`
included), so the six original runtime files stay byte-identical.
- Reference editing (Image to Image): the source is fitted to 1024 square,
encoded to the mean of the 64 moment channels, 2×2-patchified, BN-normalized
and packed into 4096 tokens appended to the image stream at RoPE time 10 —
Diffusers' `Flux2KleinPipeline` layout (`_prepare_image_ids`, scale 10). The
transformer output for reference tokens is dropped. Denoising still starts
from noise, so there is no strength control.
- Scheduling: positive Klein flow velocity, Euler updates with the ComfyUI
**10,000-value Simple grid**, exponential shift **mu 2.02**. The terminal
zero performs an Euler update; it does not invoke the transformer.
Ordinary INT4 is a separate profile from Bonsai's ternary INT2/group128 format.
Neither artifact can be loaded through the other adapter. This profile fixes
native 1024 square; it does not enable a size picker, negative prompts, Hires
Fix, Face Detail or Inpaint. SD/SDXL retain their existing capabilities.
### VAE projections in this AIO
The AIO stores both learned 1×1 VAE projections in Black Forest Labs' nested
layout — `vae.encoder.quant_conv.*` and `vae.decoder.post_quant_conv.*` — not
as Diffusers' top-level `quant_conv` / `post_quant_conv` (Diffusers'
`convert_ldm_vae_checkpoint` does not map them either). The first conversion
looked up only the top-level decoder key and therefore restores the decoder
projection from the matching official VAE. On 2026-10-02 the nested AIO
projections, plus the checked encoder and decoder weights, were compared with
that file: every FP32 AIO value rounds exactly to the official BF16 value. The
restored projection is the same learned tensor, so the published decoder is
valid; the decoder file also keeps the AIO's unused nested copy. An identity
projection is invalid and produces severe mosaic. The encoder export uses the
AIO's own FP32 `encoder.quant_conv`.
The decoder projection is restored from:
- Repository: `black-forest-labs/FLUX.2-klein-4B`.
- Revision: `e7b7dc27f91deacad38e78976d1f2b499d76a294`.
- File: `vae/diffusion_pytorch_model.safetensors` (168,120,878 bytes).
- SHA-256: `ca70d2202afe6415bdbcb8793ba8cd99fd159cfe6192381504d6c4d3036e0f04`.
The source AIO's decoder and BN statistics remain intact at FP32 precision.
The official BF16 projection is cast to FP32; this does not recover extra
precision. Both source hashes are embedded in conversion metadata. Never
substitute an unrelated VAE or an identity post-quant projection. See
`validation/vae-repair.json` and `validation/bf16-dit-reference-1024.png`.
## Reproduce conversion on an Apple Silicon Mac
Commands below run from the project root. Use a task-specific scratch directory;
keep the user's original checkpoint unchanged. Allow enough disk space for the
16.13 GB source, 4.56 GB output, environment and build products.
### 1. Python environment and pinned base resources
```sh
KLEIN_WORK="$(mktemp -d /tmp/localmuse-klein.XXXXXX)"
python3.11 -m venv "$KLEIN_WORK/env"
"$KLEIN_WORK/env/bin/python" -m pip install -r Tools/FluxKlein/requirements.txt
"$KLEIN_WORK/env/bin/python" - "$KLEIN_WORK/base" <<'PY'
import sys
from huggingface_hub import snapshot_download
snapshot_download(
repo_id='black-forest-labs/FLUX.2-klein-4B',
revision='e7b7dc27f91deacad38e78976d1f2b499d76a294',
allow_patterns=['tokenizer/*', 'text_encoder/config.json',
'transformer/config.json', 'vae/config.json',
'vae/diffusion_pytorch_model.safetensors', 'LICENSE'],
local_dir=sys.argv[1],
)
PY
```
The downloaded source file is
`~/Downloads/flux2KleinAIO_flux24BDistilledAIO_full_bf16.safetensors`:
16,132,318,096 bytes, SHA-256
`f9f60d53bb21970e19d31d85eefb294f9b0b5e36669ec81ba2bf52f102a7679e`.
The converter checks the entire source file and supplemental VAE before writing.
### 2. Stream and quantize
```sh
"$KLEIN_WORK/env/bin/python" Tools/FluxKlein/convert_klein_aio.py \
--checkpoint "$HOME/Downloads/flux2KleinAIO_flux24BDistilledAIO_full_bf16.safetensors" \
--expected-sha256 f9f60d53bb21970e19d31d85eefb294f9b0b5e36669ec81ba2bf52f102a7679e \
--tokenizer "$KLEIN_WORK/base/tokenizer" \
--vae-base "$KLEIN_WORK/base/vae/diffusion_pytorch_model.safetensors" \
--expected-vae-sha256 ca70d2202afe6415bdbcb8793ba8cd99fd159cfe6192381504d6c4d3036e0f04 \
--output "$KLEIN_WORK/converted"
```
The converter reuses the streaming safetensors writer from `Tools/ZImage` without
changing the Z-Image converter. It maps the source FLUX names, splits QKV and
swaps the final adaLN scale/shift halves to match Diffusers. It processes one
matrix at a time; loading the entire BF16 checkpoint is unnecessary. Export
is deterministic with the pinned MLX/Python versions. Compare the resulting
seven-file manifest with `Tools/FluxKlein/manifest.json` before publishing. Direct-file catalog entries must leave `downloadURL` and `checksum` null; each file carries its own URL and hash in `downloadFiles`. Mixing archive metadata into a direct manifest is rejected by the catalog validator.
Required paths:
```text
runtime_config.json
text_encoder-mlx-4bit/model.safetensors
tokenizer/tokenizer.json
tokenizer/tokenizer_config.json
transformer-mlx-4bit/model.safetensors
vae.safetensors
vae_encoder.safetensors
```
### 3. Build validation CLI and run native inference
Use Xcode's macOS executable scheme so MLX's Metal library is bundled. Keep this
package's source-package cache **separate** from the application's cache: sharing
`-clonedSourcePackagesDirPath` between different package graphs can prune the
app's `llama.xcframework` binary artifact. If that occurred, resolve the **app**
package graph and clean the app's build products before rebuilding.
```sh
(
cd Packages/bonsai-runtime
xcodebuild build -scheme klein-validate -destination 'platform=macOS' \
-derivedDataPath "$KLEIN_WORK/runtime-build" \
-clonedSourcePackagesDirPath "$KLEIN_WORK/SourcePackages"
)
MTL_DEBUG_LAYER=1 "$KLEIN_WORK/runtime-build/Build/Products/Debug/klein-validate" \
"$KLEIN_WORK/converted" "$KLEIN_WORK/native-1024.png" 1024x1024 1729 4 \
'Candid photograph of an adult woman in a cream sweater by a studio window, soft evening light, natural skin texture, analog grain.'
```
Reference editing (Image to Image) uses the same command with `--reference`;
the image is aspect-filled and center-cropped to the output size like the app:
```sh
MTL_DEBUG_LAYER=1 "$KLEIN_WORK/runtime-build/Build/Products/Debug/klein-validate" \
--reference Tools/FluxKlein/validation/native-1024.png \
"$KLEIN_WORK/converted" "$KLEIN_WORK/native-1024-edit.png" 1024x1024 1729 4 \
'Change the cream sweater to a bright red wool sweater, keep the face, pose, window and lighting unchanged.'
```
The CLI also saves parameters and the final CPU preview. It uses the same library
as the iOS app, authenticates the runtime configuration, and validates all tensor
shapes and dtypes before evaluating a graph. Encoder, DiT and VAE load sequentially.
The decoder uses overlapping 384-pixel windows with CPU cosine blending to bound
memory. Tile-local VAE statistics and attention can cause small differences from
one-shot full-frame decoding.
### 4. Independent Transformers/Diffusers comparison
```sh
MTL_DEBUG_LAYER=1 "$KLEIN_WORK/runtime-build/Build/Products/Debug/klein-validate" \
--probe --reference Tools/FluxKlein/validation/native-1024.png \
"$KLEIN_WORK/converted" "$KLEIN_WORK/probe" 256x256 1729 4 \
'Candid photograph of an adult woman in a cream sweater by a studio window, soft evening light, natural skin texture, analog grain.'
"$KLEIN_WORK/env/bin/python" Tools/FluxKlein/validate_reference.py \
--checkpoint "$HOME/Downloads/flux2KleinAIO_flux24BDistilledAIO_full_bf16.safetensors" \
--base "$KLEIN_WORK/base" --converted "$KLEIN_WORK/converted" \
--probe "$KLEIN_WORK/probe" --report "$KLEIN_WORK/reference.json" \
--prompt 'Candid photograph of an adult woman in a cream sweater by a studio window, soft evening light, natural skin texture, analog grain.'
```
The reference streams source modules on CPU, using independent Transformers and
Diffusers implementations. It compares source BF16 with native INT4, then repeats
the DiT with an independent NumPy unpacking of the quantized matrices to separate
quantization loss from Swift-port error. The VAE comparison uses the original AIO
FP32 decoder and the same restored official projection. It also verifies exact
Swift/Python token IDs and unchanged BN tensors.
Saved results in `validation/reference.json`:
| Comparison | Relative L2 | Cosine |
| --- | ---: | ---: |
| Source BF16 Qwen3 vs Swift INT4 | 0.103519 | 0.995114 |
| Source BF16 DiT vs Swift INT4, same conditioning | 0.114535 | 0.993426 |
| Dequantized INT4 DiT in Diffusers vs Swift INT4 | 0.009666 | 0.999953 |
| Restored source FP32 VAE vs Swift FP32 | 0.00000255 | ~1.0 |
The probe also encodes a 256×256 reference and predicts the same noise with its
tokens appended. `validate_edit_reference.py` compares it with independent
Diffusers code: `AutoencoderKLFlux2` loaded with the AIO's FP32 VAE through
Diffusers' own mapping (the nested projections mapped explicitly), the pipeline's
static patchify, BN, packing and `_prepare_image_ids` helpers, and
`Flux2Transformer2DModel` with source BF16 and independently dequantized INT4
matrices:
```sh
"$KLEIN_WORK/env/bin/python" Tools/FluxKlein/validate_edit_reference.py \
--checkpoint "$HOME/Downloads/flux2KleinAIO_flux24BDistilledAIO_full_bf16.safetensors" \
--base "$KLEIN_WORK/base" --converted "$KLEIN_WORK/converted" \
--probe "$KLEIN_WORK/probe" --report "$KLEIN_WORK/edit-reference.json"
```
Saved results in `validation/edit-reference.json`:
| Comparison | Relative L2 | Cosine |
| --- | ---: | ---: |
| AIO FP32 encoder (Diffusers) vs Swift FP32 | 0.00000258 | ~1.0 |
| Reference tokens (Diffusers helpers) vs Swift | 0.00000258 | ~1.0 |
| Source BF16 DiT vs Swift INT4, with reference | 0.094921 | 0.995487 |
| Dequantized INT4 DiT in Diffusers vs Swift INT4, with reference | 0.010691 | 0.999943 |
| Official BF16 VAE encoder vs Swift FP32 (weight rounding only) | 0.004676 | 0.999989 |
With the reference the predicted velocity differs from the text-only prediction
by relative L2 0.822, so the reference conditions the transformer.
These are component comparisons, not proof of identical full-pipeline BF16
images. `bf16-dit-reference-1024.png` is an additional full-size source DiT
reference with the **same INT4 text conditioning and initial noise** as the native
run. A ComfyUI seed uses a different RNG implementation and need not produce the
same image as MLX. INT4 and tiled VAE decoding also prevent pixel-exact equivalence.
### 5. Schedule and regression tests
`Packages/bonsai-runtime/Tests/BonsaiRuntimeTests/Fixtures/comfy-klein-simple.json`
contains numerical output from unmodified ComfyUI functions at revision
`65787d668397d230bf5839d69a0a7239e2dad378`. To regenerate, download the external
`comfy/samplers.py` and `comfy/model_sampling.py` from that exact revision into a
scratch reference directory, then run:
```sh
"$KLEIN_WORK/env/bin/python" Tools/FluxKlein/export_schedule_fixture.py \
--references "$KLEIN_WORK/comfy-reference" \
--revision 65787d668397d230bf5839d69a0a7239e2dad378 \
--output "$KLEIN_WORK/comfy-klein-simple.json"
(
cd Packages/bonsai-runtime
MTL_DEBUG_LAYER=1 xcodebuild test -scheme bonsai-runtime-Package \
-destination 'platform=macOS' -parallel-testing-enabled NO \
-derivedDataPath "$KLEIN_WORK/runtime-build" \
-clonedSourcePackagesDirPath "$KLEIN_WORK/SourcePackages" \
-resultBundlePath "$KLEIN_WORK/package-tests.xcresult"
)
xcodebuild build-for-testing -project LocalMuse.xcodeproj -scheme LocalMuse \
-destination 'generic/platform=iOS' -parallel-testing-enabled NO
xcodebuild test -project LocalMuse.xcodeproj -scheme LocalMuse \
-destination 'platform=iOS Simulator,id=E623201E-7B34-4622-9EF5-CD4E43558F26' \
-parallel-testing-enabled NO -only-testing:LocalMuseTests
```
Reuse an existing simulator on the newest installed compatible runtime; substitute
its UUID if the listed device is unavailable. The final application run executed 490 Swift Testing tests: 488 passed and the two physical-device opt-in tests were skipped. The full scheme's XCUITests also ran with Klein in the catalog; their only failure, `testModalGalleryFitsContentInBothAppearances`, is a pre-existing modal-height check that uses no Klein code. The package run passed all 18 tests with Metal API validation enabled. A clean rerun of the native Text to Image render after the encoder work produced a byte-identical `native-1024.png` with the same 3,606,440,960-byte MLX peak. Saved test and build summaries are in `validation/`; physical-device tests require an explicit opt-in and are skipped in normal regression runs.
## Publication and catalog installation
The source was authenticated, converted and validated before publication. The
six original runtime files were published at `c2abb5051572bcda02841b3e24af3e3fe9911d57`;
`vae_encoder.safetensors` and the seven-file manifest were added at
`646b51828f39b2541fbfcc643d6984d199c6f6a5` without changing those six files. Every remote byte size and
LFS SHA-256 was compared with the local manifest. Small files were downloaded
and hashed; each pinned public URL was also checked without authorization.
Full evidence: `verification.json`.
When replacing an artifact or publishing a new checkpoint:
1. Validate its source license, actual architecture and complete checkpoint hash.
2. Convert in a new directory and finish native/reference checks.
3. Authenticate to HF using `hf auth login`; keep credentials out of scripts/logs.
4. Create a private repository, upload the runtime files plus model card, license,
notice, source metadata and manifest with `huggingface_hub.HfApi.upload_folder`.
5. Verify all remote sizes/hashes against the local manifest, publish it, and
verify anonymous access to the resulting immutable commit.
6. Put **that commit** and every exact size and SHA-256 hash in the catalog.
Never use `main` in an application download manifest.
7. Add the preview asset **and** the allowlist entry in `ModelIdentityIcon`.
Match metadata in `PreviewCatalog` and add profile/catalog/lifecycle tests.
Current catalog version is **60**; the 24th image model uses this manifest and
profile. The default image model and language catalog remain unchanged. Model
selection is lazy and all generation shares `MLExecutionGate`. Cancellation
keeps synchronous MLX work registered until it unwinds. The runtime checks
cancellation and the foreground gate before each encoder layer, transformer
block and VAE operation. The 64 MiB MLX cache limit is restored at exit.
Preparation reports real stages: text encoder and image transformer for Text to
Image, plus the image encoder between them for Image to Image. Safetensors load
lazily, so the transformer weights are evaluated inside their stage. The VAE
decoder is loaded only at decoding after transformer eviction. Optional CPU clean-latent
previews use the existing request-scoped relay and filtering limits (128 pixels,
or 12 protected pixels), without a new neural network or changes to final latents.
## Validation and manual iPhone check
Native Swift **1024 × 1024 / four-step** inference on an **M1 Pro, 16 GB Mac**:
- Elapsed: **99.47 seconds**, with Metal API validation enabled.
- Peak MLX allocation: **3,606,375,424 bytes** (3439.3 MiB / 3.36 GiB).
- Preparation counts: `[0, 1, 2]`; four live preview frames, 128 × 128.
- Native image: `validation/native-1024.png`; parameters and preview saved beside it.
Reference editing at **1024 × 1024 / four steps** on the same Mac
(`validation/native-1024-edit.png`, source `native-1024.png`):
- Elapsed: **173.81 seconds**, with Metal API validation enabled.
- Peak MLX allocation: **4,349,946,880 bytes** (4148.4 MiB / 4.05 GiB); the
4096 reference tokens lengthen every transformer step.
- Preparation counts: `[0, 1, 2, 3]`; four live preview frames, 128 × 128.
These are Mac measurements, **not iPhone speed or total-process-memory claims**.
The physical-device check is left to the user. The current 12 GiB catalog floor
is conservative; do not enable 8 GB devices based on host allocations alone.
For normal manual testing, build/run LocalMuse, download **FLUX.2 Klein 4B**,
keep the default 1024 square / four steps / CFG 1, and use the neutral English
prompt above. Check both model-preparation stages, four denoising previews,
correct final dimensions and image quality. Then open Image to Image with that
result as the source and the edit prompt above: there is no strength control,
the prompt is required, preparation shows three stages (text encoder, image
encoder, image transformer) and only the requested change should differ. Also
check cancellation, returning from the background, and switching back to an
existing SD/SDXL or Z-Image model.
A developer-only `KleinDeviceTests.installedCheckpointGeneratesAndEditsNativeCanvasWithMetalValidation()`
smoke test is available with `LOCALMUSE_KLEIN_DEVICE_QA=1` and `MTL_DEBUG_LAYER=1`.
It renders the prompt above, then reference-edits that render (`klein-device-qa`
and `klein-device-qa-edit` PNG/preview/JSON in Caches, including peak MLX bytes).
It uses the installed authenticated model or an explicitly staged cache
directory and does not write gallery or persistence records. It is not enabled
or run automatically.
## Cleanup
Preserve source checkpoints in Downloads and these repository files. After
publication and evidence checks, remove only the task's scratch environment,
converted duplicate, external references, validation arrays and build products.
Do not delete installed app models, user gallery data, shared HF caches or the
app's existing SourcePackages cache. This run's temporary package `.build`
directory and generated `default.profraw` can also be removed after validation.