mlx-flux2-klein-4b-aio-4bit / conversion-guide.md
Devserhii's picture
Document Image to Image reference editing and verify runtime commit 646b518
cde0259 verified
|
Raw History Blame Contribute Delete
21 kB

FLUX.2 Klein 4B AIO: conversion and LocalMuse integration

Published model

Field Value
App name / ID FLUX.2 Klein 4B / flux2_klein_4b_aio_mlx
Source Civitai model 2327389, version 2618128, BF16 file 2511488
Source version Flux.2-4B-Distilled-AIO, repack by SeeSeeLP / SeeSee21
HF repository LocalMuseAI/mlx-flux2-klein-4b-aio-4bit
Immutable runtime commit 646b51828f39b2541fbfcc643d6984d199c6f6a5 (first six files unchanged since c2abb5051572bcda02841b3e24af3e3fe9911d57)
Runtime / backend flux2Klein4B4Bit / flux2KleinMLX
Payload 4,560,019,137 bytes (4.56 GB / 4.25 GiB), seven authenticated files
Defaults 1024 × 1024, 4 steps, CFG 1, Euler / Simple
Allowed steps 4–6
Workflows Text to Image; Image to Image as reference editing. One image per request
Device floor 12 GiB RAM; physical iPhone validation pending

This is an AIO repack of the official step-distilled Klein 4B, not the 9B model or the undistilled base. The author's recommendations are 4–6 steps, CFG 1, Euler, Simple or normal scheduling, and native 1024 square. LocalMuse implements the Simple variant. Existing Bonsai, Z-Image, SD and SDXL catalog entries retain their previous IDs, manifests, resolutions and defaults.

The model, Qwen3 text encoder and official VAE are Apache-2.0. The source card permits redistribution and derivatives; exact metadata is in source.json. Preserve LICENSE, NOTICE, the author attribution and the BFL/Qwen notices when redistributing converted weights. Runtime code adapted from Duration AI retains its Apache license and complete upstream NOTICE. ComfyUI is only an external development reference; no ComfyUI implementation is included in the app.

What is converted

  • DiT: 3072 hidden dimensions, 24 heads, five double-stream and twenty single-stream blocks. Attention and FFN matrices use ordinary affine INT4, group 64; sensitive input/output, timestep and modulation layers remain BF16.
  • Qwen3-4B: INT4/group64 including row-wise embedding lookup. Raw hidden states after layers 9, 18 and 27 are concatenated to 7680 channels. Layers after the last tap, the final language normalization and language head are unused and omitted. No entire embedding-table dequantization occurs at runtime.
  • Tokenizer: official BFL-pinned Qwen3 files, empty thinking chat template, right padding to 512 tokens, causal and padding attention masks.
  • VAE: original 32-channel AIO decoder, encoder and packed 128-channel latent BN statistics retained in FP32; see the projection note below. Convolution weights change layout to MLX OHWI; attention weights become 2D. The encoder is a separate vae_encoder.safetensors (its nested quant_conv included), so the six original runtime files stay byte-identical.
  • Reference editing (Image to Image): the source is fitted to 1024 square, encoded to the mean of the 64 moment channels, 2×2-patchified, BN-normalized and packed into 4096 tokens appended to the image stream at RoPE time 10 — Diffusers' Flux2KleinPipeline layout (_prepare_image_ids, scale 10). The transformer output for reference tokens is dropped. Denoising still starts from noise, so there is no strength control.
  • Scheduling: positive Klein flow velocity, Euler updates with the ComfyUI 10,000-value Simple grid, exponential shift mu 2.02. The terminal zero performs an Euler update; it does not invoke the transformer.

Ordinary INT4 is a separate profile from Bonsai's ternary INT2/group128 format. Neither artifact can be loaded through the other adapter. This profile fixes native 1024 square; it does not enable a size picker, negative prompts, Hires Fix, Face Detail or Inpaint. SD/SDXL retain their existing capabilities.

VAE projections in this AIO

The AIO stores both learned 1×1 VAE projections in Black Forest Labs' nested layout — vae.encoder.quant_conv.* and vae.decoder.post_quant_conv.* — not as Diffusers' top-level quant_conv / post_quant_conv (Diffusers' convert_ldm_vae_checkpoint does not map them either). The first conversion looked up only the top-level decoder key and therefore restores the decoder projection from the matching official VAE. On 2026-10-02 the nested AIO projections, plus the checked encoder and decoder weights, were compared with that file: every FP32 AIO value rounds exactly to the official BF16 value. The restored projection is the same learned tensor, so the published decoder is valid; the decoder file also keeps the AIO's unused nested copy. An identity projection is invalid and produces severe mosaic. The encoder export uses the AIO's own FP32 encoder.quant_conv.

The decoder projection is restored from:

  • Repository: black-forest-labs/FLUX.2-klein-4B.
  • Revision: e7b7dc27f91deacad38e78976d1f2b499d76a294.
  • File: vae/diffusion_pytorch_model.safetensors (168,120,878 bytes).
  • SHA-256: ca70d2202afe6415bdbcb8793ba8cd99fd159cfe6192381504d6c4d3036e0f04.

The source AIO's decoder and BN statistics remain intact at FP32 precision. The official BF16 projection is cast to FP32; this does not recover extra precision. Both source hashes are embedded in conversion metadata. Never substitute an unrelated VAE or an identity post-quant projection. See validation/vae-repair.json and validation/bf16-dit-reference-1024.png.

Reproduce conversion on an Apple Silicon Mac

Commands below run from the project root. Use a task-specific scratch directory; keep the user's original checkpoint unchanged. Allow enough disk space for the 16.13 GB source, 4.56 GB output, environment and build products.

1. Python environment and pinned base resources

KLEIN_WORK="$(mktemp -d /tmp/localmuse-klein.XXXXXX)"
python3.11 -m venv "$KLEIN_WORK/env"
"$KLEIN_WORK/env/bin/python" -m pip install -r Tools/FluxKlein/requirements.txt
"$KLEIN_WORK/env/bin/python" - "$KLEIN_WORK/base" <<'PY'
import sys
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id='black-forest-labs/FLUX.2-klein-4B',
    revision='e7b7dc27f91deacad38e78976d1f2b499d76a294',
    allow_patterns=['tokenizer/*', 'text_encoder/config.json',
                    'transformer/config.json', 'vae/config.json',
                    'vae/diffusion_pytorch_model.safetensors', 'LICENSE'],
    local_dir=sys.argv[1],
)
PY

The downloaded source file is ~/Downloads/flux2KleinAIO_flux24BDistilledAIO_full_bf16.safetensors: 16,132,318,096 bytes, SHA-256 f9f60d53bb21970e19d31d85eefb294f9b0b5e36669ec81ba2bf52f102a7679e. The converter checks the entire source file and supplemental VAE before writing.

2. Stream and quantize

"$KLEIN_WORK/env/bin/python" Tools/FluxKlein/convert_klein_aio.py \
  --checkpoint "$HOME/Downloads/flux2KleinAIO_flux24BDistilledAIO_full_bf16.safetensors" \
  --expected-sha256 f9f60d53bb21970e19d31d85eefb294f9b0b5e36669ec81ba2bf52f102a7679e \
  --tokenizer "$KLEIN_WORK/base/tokenizer" \
  --vae-base "$KLEIN_WORK/base/vae/diffusion_pytorch_model.safetensors" \
  --expected-vae-sha256 ca70d2202afe6415bdbcb8793ba8cd99fd159cfe6192381504d6c4d3036e0f04 \
  --output "$KLEIN_WORK/converted"

The converter reuses the streaming safetensors writer from Tools/ZImage without changing the Z-Image converter. It maps the source FLUX names, splits QKV and swaps the final adaLN scale/shift halves to match Diffusers. It processes one matrix at a time; loading the entire BF16 checkpoint is unnecessary. Export is deterministic with the pinned MLX/Python versions. Compare the resulting seven-file manifest with Tools/FluxKlein/manifest.json before publishing. Direct-file catalog entries must leave downloadURL and checksum null; each file carries its own URL and hash in downloadFiles. Mixing archive metadata into a direct manifest is rejected by the catalog validator.

Required paths:

runtime_config.json
text_encoder-mlx-4bit/model.safetensors
tokenizer/tokenizer.json
tokenizer/tokenizer_config.json
transformer-mlx-4bit/model.safetensors
vae.safetensors
vae_encoder.safetensors

3. Build validation CLI and run native inference

Use Xcode's macOS executable scheme so MLX's Metal library is bundled. Keep this package's source-package cache separate from the application's cache: sharing -clonedSourcePackagesDirPath between different package graphs can prune the app's llama.xcframework binary artifact. If that occurred, resolve the app package graph and clean the app's build products before rebuilding.

(
  cd Packages/bonsai-runtime
  xcodebuild build -scheme klein-validate -destination 'platform=macOS' \
    -derivedDataPath "$KLEIN_WORK/runtime-build" \
    -clonedSourcePackagesDirPath "$KLEIN_WORK/SourcePackages"
)
MTL_DEBUG_LAYER=1 "$KLEIN_WORK/runtime-build/Build/Products/Debug/klein-validate" \
  "$KLEIN_WORK/converted" "$KLEIN_WORK/native-1024.png" 1024x1024 1729 4 \
  'Candid photograph of an adult woman in a cream sweater by a studio window, soft evening light, natural skin texture, analog grain.'

Reference editing (Image to Image) uses the same command with --reference; the image is aspect-filled and center-cropped to the output size like the app:

MTL_DEBUG_LAYER=1 "$KLEIN_WORK/runtime-build/Build/Products/Debug/klein-validate" \
  --reference Tools/FluxKlein/validation/native-1024.png \
  "$KLEIN_WORK/converted" "$KLEIN_WORK/native-1024-edit.png" 1024x1024 1729 4 \
  'Change the cream sweater to a bright red wool sweater, keep the face, pose, window and lighting unchanged.'

The CLI also saves parameters and the final CPU preview. It uses the same library as the iOS app, authenticates the runtime configuration, and validates all tensor shapes and dtypes before evaluating a graph. Encoder, DiT and VAE load sequentially. The decoder uses overlapping 384-pixel windows with CPU cosine blending to bound memory. Tile-local VAE statistics and attention can cause small differences from one-shot full-frame decoding.

4. Independent Transformers/Diffusers comparison

MTL_DEBUG_LAYER=1 "$KLEIN_WORK/runtime-build/Build/Products/Debug/klein-validate" \
  --probe --reference Tools/FluxKlein/validation/native-1024.png \
  "$KLEIN_WORK/converted" "$KLEIN_WORK/probe" 256x256 1729 4 \
  'Candid photograph of an adult woman in a cream sweater by a studio window, soft evening light, natural skin texture, analog grain.'
"$KLEIN_WORK/env/bin/python" Tools/FluxKlein/validate_reference.py \
  --checkpoint "$HOME/Downloads/flux2KleinAIO_flux24BDistilledAIO_full_bf16.safetensors" \
  --base "$KLEIN_WORK/base" --converted "$KLEIN_WORK/converted" \
  --probe "$KLEIN_WORK/probe" --report "$KLEIN_WORK/reference.json" \
  --prompt 'Candid photograph of an adult woman in a cream sweater by a studio window, soft evening light, natural skin texture, analog grain.'

The reference streams source modules on CPU, using independent Transformers and Diffusers implementations. It compares source BF16 with native INT4, then repeats the DiT with an independent NumPy unpacking of the quantized matrices to separate quantization loss from Swift-port error. The VAE comparison uses the original AIO FP32 decoder and the same restored official projection. It also verifies exact Swift/Python token IDs and unchanged BN tensors.

Saved results in validation/reference.json:

Comparison Relative L2 Cosine
Source BF16 Qwen3 vs Swift INT4 0.103519 0.995114
Source BF16 DiT vs Swift INT4, same conditioning 0.114535 0.993426
Dequantized INT4 DiT in Diffusers vs Swift INT4 0.009666 0.999953
Restored source FP32 VAE vs Swift FP32 0.00000255 ~1.0

The probe also encodes a 256×256 reference and predicts the same noise with its tokens appended. validate_edit_reference.py compares it with independent Diffusers code: AutoencoderKLFlux2 loaded with the AIO's FP32 VAE through Diffusers' own mapping (the nested projections mapped explicitly), the pipeline's static patchify, BN, packing and _prepare_image_ids helpers, and Flux2Transformer2DModel with source BF16 and independently dequantized INT4 matrices:

"$KLEIN_WORK/env/bin/python" Tools/FluxKlein/validate_edit_reference.py \
  --checkpoint "$HOME/Downloads/flux2KleinAIO_flux24BDistilledAIO_full_bf16.safetensors" \
  --base "$KLEIN_WORK/base" --converted "$KLEIN_WORK/converted" \
  --probe "$KLEIN_WORK/probe" --report "$KLEIN_WORK/edit-reference.json"

Saved results in validation/edit-reference.json:

Comparison Relative L2 Cosine
AIO FP32 encoder (Diffusers) vs Swift FP32 0.00000258 ~1.0
Reference tokens (Diffusers helpers) vs Swift 0.00000258 ~1.0
Source BF16 DiT vs Swift INT4, with reference 0.094921 0.995487
Dequantized INT4 DiT in Diffusers vs Swift INT4, with reference 0.010691 0.999943
Official BF16 VAE encoder vs Swift FP32 (weight rounding only) 0.004676 0.999989

With the reference the predicted velocity differs from the text-only prediction by relative L2 0.822, so the reference conditions the transformer.

These are component comparisons, not proof of identical full-pipeline BF16 images. bf16-dit-reference-1024.png is an additional full-size source DiT reference with the same INT4 text conditioning and initial noise as the native run. A ComfyUI seed uses a different RNG implementation and need not produce the same image as MLX. INT4 and tiled VAE decoding also prevent pixel-exact equivalence.

5. Schedule and regression tests

Packages/bonsai-runtime/Tests/BonsaiRuntimeTests/Fixtures/comfy-klein-simple.json contains numerical output from unmodified ComfyUI functions at revision 65787d668397d230bf5839d69a0a7239e2dad378. To regenerate, download the external comfy/samplers.py and comfy/model_sampling.py from that exact revision into a scratch reference directory, then run:

"$KLEIN_WORK/env/bin/python" Tools/FluxKlein/export_schedule_fixture.py \
  --references "$KLEIN_WORK/comfy-reference" \
  --revision 65787d668397d230bf5839d69a0a7239e2dad378 \
  --output "$KLEIN_WORK/comfy-klein-simple.json"
(
  cd Packages/bonsai-runtime
  MTL_DEBUG_LAYER=1 xcodebuild test -scheme bonsai-runtime-Package \
    -destination 'platform=macOS' -parallel-testing-enabled NO \
    -derivedDataPath "$KLEIN_WORK/runtime-build" \
    -clonedSourcePackagesDirPath "$KLEIN_WORK/SourcePackages" \
    -resultBundlePath "$KLEIN_WORK/package-tests.xcresult"
)
xcodebuild build-for-testing -project LocalMuse.xcodeproj -scheme LocalMuse \
  -destination 'generic/platform=iOS' -parallel-testing-enabled NO
xcodebuild test -project LocalMuse.xcodeproj -scheme LocalMuse \
  -destination 'platform=iOS Simulator,id=E623201E-7B34-4622-9EF5-CD4E43558F26' \
  -parallel-testing-enabled NO -only-testing:LocalMuseTests

Reuse an existing simulator on the newest installed compatible runtime; substitute its UUID if the listed device is unavailable. The final application run executed 490 Swift Testing tests: 488 passed and the two physical-device opt-in tests were skipped. The full scheme's XCUITests also ran with Klein in the catalog; their only failure, testModalGalleryFitsContentInBothAppearances, is a pre-existing modal-height check that uses no Klein code. The package run passed all 18 tests with Metal API validation enabled. A clean rerun of the native Text to Image render after the encoder work produced a byte-identical native-1024.png with the same 3,606,440,960-byte MLX peak. Saved test and build summaries are in validation/; physical-device tests require an explicit opt-in and are skipped in normal regression runs.

Publication and catalog installation

The source was authenticated, converted and validated before publication. The six original runtime files were published at c2abb5051572bcda02841b3e24af3e3fe9911d57; vae_encoder.safetensors and the seven-file manifest were added at 646b51828f39b2541fbfcc643d6984d199c6f6a5 without changing those six files. Every remote byte size and LFS SHA-256 was compared with the local manifest. Small files were downloaded and hashed; each pinned public URL was also checked without authorization. Full evidence: verification.json.

When replacing an artifact or publishing a new checkpoint:

  1. Validate its source license, actual architecture and complete checkpoint hash.
  2. Convert in a new directory and finish native/reference checks.
  3. Authenticate to HF using hf auth login; keep credentials out of scripts/logs.
  4. Create a private repository, upload the runtime files plus model card, license, notice, source metadata and manifest with huggingface_hub.HfApi.upload_folder.
  5. Verify all remote sizes/hashes against the local manifest, publish it, and verify anonymous access to the resulting immutable commit.
  6. Put that commit and every exact size and SHA-256 hash in the catalog. Never use main in an application download manifest.
  7. Add the preview asset and the allowlist entry in ModelIdentityIcon. Match metadata in PreviewCatalog and add profile/catalog/lifecycle tests.

Current catalog version is 60; the 24th image model uses this manifest and profile. The default image model and language catalog remain unchanged. Model selection is lazy and all generation shares MLExecutionGate. Cancellation keeps synchronous MLX work registered until it unwinds. The runtime checks cancellation and the foreground gate before each encoder layer, transformer block and VAE operation. The 64 MiB MLX cache limit is restored at exit.

Preparation reports real stages: text encoder and image transformer for Text to Image, plus the image encoder between them for Image to Image. Safetensors load lazily, so the transformer weights are evaluated inside their stage. The VAE decoder is loaded only at decoding after transformer eviction. Optional CPU clean-latent previews use the existing request-scoped relay and filtering limits (128 pixels, or 12 protected pixels), without a new neural network or changes to final latents.

Validation and manual iPhone check

Native Swift 1024 × 1024 / four-step inference on an M1 Pro, 16 GB Mac:

  • Elapsed: 99.47 seconds, with Metal API validation enabled.
  • Peak MLX allocation: 3,606,375,424 bytes (3439.3 MiB / 3.36 GiB).
  • Preparation counts: [0, 1, 2]; four live preview frames, 128 × 128.
  • Native image: validation/native-1024.png; parameters and preview saved beside it.

Reference editing at 1024 × 1024 / four steps on the same Mac (validation/native-1024-edit.png, source native-1024.png):

  • Elapsed: 173.81 seconds, with Metal API validation enabled.
  • Peak MLX allocation: 4,349,946,880 bytes (4148.4 MiB / 4.05 GiB); the 4096 reference tokens lengthen every transformer step.
  • Preparation counts: [0, 1, 2, 3]; four live preview frames, 128 × 128.

These are Mac measurements, not iPhone speed or total-process-memory claims. The physical-device check is left to the user. The current 12 GiB catalog floor is conservative; do not enable 8 GB devices based on host allocations alone.

For normal manual testing, build/run LocalMuse, download FLUX.2 Klein 4B, keep the default 1024 square / four steps / CFG 1, and use the neutral English prompt above. Check both model-preparation stages, four denoising previews, correct final dimensions and image quality. Then open Image to Image with that result as the source and the edit prompt above: there is no strength control, the prompt is required, preparation shows three stages (text encoder, image encoder, image transformer) and only the requested change should differ. Also check cancellation, returning from the background, and switching back to an existing SD/SDXL or Z-Image model.

A developer-only KleinDeviceTests.installedCheckpointGeneratesAndEditsNativeCanvasWithMetalValidation() smoke test is available with LOCALMUSE_KLEIN_DEVICE_QA=1 and MTL_DEBUG_LAYER=1. It renders the prompt above, then reference-edits that render (klein-device-qa and klein-device-qa-edit PNG/preview/JSON in Caches, including peak MLX bytes). It uses the installed authenticated model or an explicitly staged cache directory and does not write gallery or persistence records. It is not enabled or run automatically.

Cleanup

Preserve source checkpoints in Downloads and these repository files. After publication and evidence checks, remove only the task's scratch environment, converted duplicate, external references, validation arrays and build products. Do not delete installed app models, user gallery data, shared HF caches or the app's existing SourcePackages cache. This run's temporary package .build directory and generated default.profraw can also be removed after validation.