# FLUX.2 Klein 4B AIO: conversion and LocalMuse integration ## Published model | Field | Value | | --- | --- | | App name / ID | **FLUX.2 Klein 4B** / `flux2_klein_4b_aio_mlx` | | Source | [Civitai model 2327389, version 2618128](https://civitai.com/models/2327389?modelVersionId=2618128), BF16 file 2511488 | | Source version | `Flux.2-4B-Distilled-AIO`, repack by SeeSeeLP / SeeSee21 | | HF repository | [LocalMuseAI/mlx-flux2-klein-4b-aio-4bit](https://huggingface.co/LocalMuseAI/mlx-flux2-klein-4b-aio-4bit) | | Immutable runtime commit | `646b51828f39b2541fbfcc643d6984d199c6f6a5` (first six files unchanged since `c2abb5051572bcda02841b3e24af3e3fe9911d57`) | | Runtime / backend | `flux2Klein4B4Bit` / `flux2KleinMLX` | | Payload | **4,560,019,137 bytes** (4.56 GB / 4.25 GiB), seven authenticated files | | Defaults | **1024 × 1024, 4 steps, CFG 1, Euler / Simple** | | Allowed steps | 4–6 | | Workflows | Text to Image; Image to Image as reference editing. One image per request | | Device floor | 12 GiB RAM; physical iPhone validation pending | This is an AIO repack of the official step-distilled **Klein 4B**, not the 9B model or the undistilled base. The author's recommendations are 4–6 steps, CFG 1, Euler, Simple or normal scheduling, and native 1024 square. LocalMuse implements the Simple variant. Existing Bonsai, Z-Image, SD and SDXL catalog entries retain their previous IDs, manifests, resolutions and defaults. The model, Qwen3 text encoder and official VAE are Apache-2.0. The source card permits redistribution and derivatives; exact metadata is in `source.json`. Preserve `LICENSE`, `NOTICE`, the author attribution and the BFL/Qwen notices when redistributing converted weights. Runtime code adapted from Duration AI retains its Apache license and complete upstream NOTICE. ComfyUI is only an external development reference; no ComfyUI implementation is included in the app. ## What is converted - DiT: 3072 hidden dimensions, 24 heads, five double-stream and twenty single-stream blocks. Attention and FFN matrices use ordinary affine INT4, group 64; sensitive input/output, timestep and modulation layers remain BF16. - Qwen3-4B: INT4/group64 including row-wise embedding lookup. Raw hidden states after layers **9, 18 and 27** are concatenated to 7680 channels. Layers after the last tap, the final language normalization and language head are unused and omitted. No entire embedding-table dequantization occurs at runtime. - Tokenizer: official BFL-pinned Qwen3 files, empty thinking chat template, right padding to 512 tokens, causal and padding attention masks. - VAE: original **32-channel** AIO decoder, encoder and packed 128-channel latent BN statistics retained in **FP32**; see the projection note below. Convolution weights change layout to MLX OHWI; attention weights become 2D. The encoder is a separate `vae_encoder.safetensors` (its nested `quant_conv` included), so the six original runtime files stay byte-identical. - Reference editing (Image to Image): the source is fitted to 1024 square, encoded to the mean of the 64 moment channels, 2×2-patchified, BN-normalized and packed into 4096 tokens appended to the image stream at RoPE time 10 — Diffusers' `Flux2KleinPipeline` layout (`_prepare_image_ids`, scale 10). The transformer output for reference tokens is dropped. Denoising still starts from noise, so there is no strength control. - Scheduling: positive Klein flow velocity, Euler updates with the ComfyUI **10,000-value Simple grid**, exponential shift **mu 2.02**. The terminal zero performs an Euler update; it does not invoke the transformer. Ordinary INT4 is a separate profile from Bonsai's ternary INT2/group128 format. Neither artifact can be loaded through the other adapter. This profile fixes native 1024 square; it does not enable a size picker, negative prompts, Hires Fix, Face Detail or Inpaint. SD/SDXL retain their existing capabilities. ### VAE projections in this AIO The AIO stores both learned 1×1 VAE projections in Black Forest Labs' nested layout — `vae.encoder.quant_conv.*` and `vae.decoder.post_quant_conv.*` — not as Diffusers' top-level `quant_conv` / `post_quant_conv` (Diffusers' `convert_ldm_vae_checkpoint` does not map them either). The first conversion looked up only the top-level decoder key and therefore restores the decoder projection from the matching official VAE. On 2026-10-02 the nested AIO projections, plus the checked encoder and decoder weights, were compared with that file: every FP32 AIO value rounds exactly to the official BF16 value. The restored projection is the same learned tensor, so the published decoder is valid; the decoder file also keeps the AIO's unused nested copy. An identity projection is invalid and produces severe mosaic. The encoder export uses the AIO's own FP32 `encoder.quant_conv`. The decoder projection is restored from: - Repository: `black-forest-labs/FLUX.2-klein-4B`. - Revision: `e7b7dc27f91deacad38e78976d1f2b499d76a294`. - File: `vae/diffusion_pytorch_model.safetensors` (168,120,878 bytes). - SHA-256: `ca70d2202afe6415bdbcb8793ba8cd99fd159cfe6192381504d6c4d3036e0f04`. The source AIO's decoder and BN statistics remain intact at FP32 precision. The official BF16 projection is cast to FP32; this does not recover extra precision. Both source hashes are embedded in conversion metadata. Never substitute an unrelated VAE or an identity post-quant projection. See `validation/vae-repair.json` and `validation/bf16-dit-reference-1024.png`. ## Reproduce conversion on an Apple Silicon Mac Commands below run from the project root. Use a task-specific scratch directory; keep the user's original checkpoint unchanged. Allow enough disk space for the 16.13 GB source, 4.56 GB output, environment and build products. ### 1. Python environment and pinned base resources ```sh KLEIN_WORK="$(mktemp -d /tmp/localmuse-klein.XXXXXX)" python3.11 -m venv "$KLEIN_WORK/env" "$KLEIN_WORK/env/bin/python" -m pip install -r Tools/FluxKlein/requirements.txt "$KLEIN_WORK/env/bin/python" - "$KLEIN_WORK/base" <<'PY' import sys from huggingface_hub import snapshot_download snapshot_download( repo_id='black-forest-labs/FLUX.2-klein-4B', revision='e7b7dc27f91deacad38e78976d1f2b499d76a294', allow_patterns=['tokenizer/*', 'text_encoder/config.json', 'transformer/config.json', 'vae/config.json', 'vae/diffusion_pytorch_model.safetensors', 'LICENSE'], local_dir=sys.argv[1], ) PY ``` The downloaded source file is `~/Downloads/flux2KleinAIO_flux24BDistilledAIO_full_bf16.safetensors`: 16,132,318,096 bytes, SHA-256 `f9f60d53bb21970e19d31d85eefb294f9b0b5e36669ec81ba2bf52f102a7679e`. The converter checks the entire source file and supplemental VAE before writing. ### 2. Stream and quantize ```sh "$KLEIN_WORK/env/bin/python" Tools/FluxKlein/convert_klein_aio.py \ --checkpoint "$HOME/Downloads/flux2KleinAIO_flux24BDistilledAIO_full_bf16.safetensors" \ --expected-sha256 f9f60d53bb21970e19d31d85eefb294f9b0b5e36669ec81ba2bf52f102a7679e \ --tokenizer "$KLEIN_WORK/base/tokenizer" \ --vae-base "$KLEIN_WORK/base/vae/diffusion_pytorch_model.safetensors" \ --expected-vae-sha256 ca70d2202afe6415bdbcb8793ba8cd99fd159cfe6192381504d6c4d3036e0f04 \ --output "$KLEIN_WORK/converted" ``` The converter reuses the streaming safetensors writer from `Tools/ZImage` without changing the Z-Image converter. It maps the source FLUX names, splits QKV and swaps the final adaLN scale/shift halves to match Diffusers. It processes one matrix at a time; loading the entire BF16 checkpoint is unnecessary. Export is deterministic with the pinned MLX/Python versions. Compare the resulting seven-file manifest with `Tools/FluxKlein/manifest.json` before publishing. Direct-file catalog entries must leave `downloadURL` and `checksum` null; each file carries its own URL and hash in `downloadFiles`. Mixing archive metadata into a direct manifest is rejected by the catalog validator. Required paths: ```text runtime_config.json text_encoder-mlx-4bit/model.safetensors tokenizer/tokenizer.json tokenizer/tokenizer_config.json transformer-mlx-4bit/model.safetensors vae.safetensors vae_encoder.safetensors ``` ### 3. Build validation CLI and run native inference Use Xcode's macOS executable scheme so MLX's Metal library is bundled. Keep this package's source-package cache **separate** from the application's cache: sharing `-clonedSourcePackagesDirPath` between different package graphs can prune the app's `llama.xcframework` binary artifact. If that occurred, resolve the **app** package graph and clean the app's build products before rebuilding. ```sh ( cd Packages/bonsai-runtime xcodebuild build -scheme klein-validate -destination 'platform=macOS' \ -derivedDataPath "$KLEIN_WORK/runtime-build" \ -clonedSourcePackagesDirPath "$KLEIN_WORK/SourcePackages" ) MTL_DEBUG_LAYER=1 "$KLEIN_WORK/runtime-build/Build/Products/Debug/klein-validate" \ "$KLEIN_WORK/converted" "$KLEIN_WORK/native-1024.png" 1024x1024 1729 4 \ 'Candid photograph of an adult woman in a cream sweater by a studio window, soft evening light, natural skin texture, analog grain.' ``` Reference editing (Image to Image) uses the same command with `--reference`; the image is aspect-filled and center-cropped to the output size like the app: ```sh MTL_DEBUG_LAYER=1 "$KLEIN_WORK/runtime-build/Build/Products/Debug/klein-validate" \ --reference Tools/FluxKlein/validation/native-1024.png \ "$KLEIN_WORK/converted" "$KLEIN_WORK/native-1024-edit.png" 1024x1024 1729 4 \ 'Change the cream sweater to a bright red wool sweater, keep the face, pose, window and lighting unchanged.' ``` The CLI also saves parameters and the final CPU preview. It uses the same library as the iOS app, authenticates the runtime configuration, and validates all tensor shapes and dtypes before evaluating a graph. Encoder, DiT and VAE load sequentially. The decoder uses overlapping 384-pixel windows with CPU cosine blending to bound memory. Tile-local VAE statistics and attention can cause small differences from one-shot full-frame decoding. ### 4. Independent Transformers/Diffusers comparison ```sh MTL_DEBUG_LAYER=1 "$KLEIN_WORK/runtime-build/Build/Products/Debug/klein-validate" \ --probe --reference Tools/FluxKlein/validation/native-1024.png \ "$KLEIN_WORK/converted" "$KLEIN_WORK/probe" 256x256 1729 4 \ 'Candid photograph of an adult woman in a cream sweater by a studio window, soft evening light, natural skin texture, analog grain.' "$KLEIN_WORK/env/bin/python" Tools/FluxKlein/validate_reference.py \ --checkpoint "$HOME/Downloads/flux2KleinAIO_flux24BDistilledAIO_full_bf16.safetensors" \ --base "$KLEIN_WORK/base" --converted "$KLEIN_WORK/converted" \ --probe "$KLEIN_WORK/probe" --report "$KLEIN_WORK/reference.json" \ --prompt 'Candid photograph of an adult woman in a cream sweater by a studio window, soft evening light, natural skin texture, analog grain.' ``` The reference streams source modules on CPU, using independent Transformers and Diffusers implementations. It compares source BF16 with native INT4, then repeats the DiT with an independent NumPy unpacking of the quantized matrices to separate quantization loss from Swift-port error. The VAE comparison uses the original AIO FP32 decoder and the same restored official projection. It also verifies exact Swift/Python token IDs and unchanged BN tensors. Saved results in `validation/reference.json`: | Comparison | Relative L2 | Cosine | | --- | ---: | ---: | | Source BF16 Qwen3 vs Swift INT4 | 0.103519 | 0.995114 | | Source BF16 DiT vs Swift INT4, same conditioning | 0.114535 | 0.993426 | | Dequantized INT4 DiT in Diffusers vs Swift INT4 | 0.009666 | 0.999953 | | Restored source FP32 VAE vs Swift FP32 | 0.00000255 | ~1.0 | The probe also encodes a 256×256 reference and predicts the same noise with its tokens appended. `validate_edit_reference.py` compares it with independent Diffusers code: `AutoencoderKLFlux2` loaded with the AIO's FP32 VAE through Diffusers' own mapping (the nested projections mapped explicitly), the pipeline's static patchify, BN, packing and `_prepare_image_ids` helpers, and `Flux2Transformer2DModel` with source BF16 and independently dequantized INT4 matrices: ```sh "$KLEIN_WORK/env/bin/python" Tools/FluxKlein/validate_edit_reference.py \ --checkpoint "$HOME/Downloads/flux2KleinAIO_flux24BDistilledAIO_full_bf16.safetensors" \ --base "$KLEIN_WORK/base" --converted "$KLEIN_WORK/converted" \ --probe "$KLEIN_WORK/probe" --report "$KLEIN_WORK/edit-reference.json" ``` Saved results in `validation/edit-reference.json`: | Comparison | Relative L2 | Cosine | | --- | ---: | ---: | | AIO FP32 encoder (Diffusers) vs Swift FP32 | 0.00000258 | ~1.0 | | Reference tokens (Diffusers helpers) vs Swift | 0.00000258 | ~1.0 | | Source BF16 DiT vs Swift INT4, with reference | 0.094921 | 0.995487 | | Dequantized INT4 DiT in Diffusers vs Swift INT4, with reference | 0.010691 | 0.999943 | | Official BF16 VAE encoder vs Swift FP32 (weight rounding only) | 0.004676 | 0.999989 | With the reference the predicted velocity differs from the text-only prediction by relative L2 0.822, so the reference conditions the transformer. These are component comparisons, not proof of identical full-pipeline BF16 images. `bf16-dit-reference-1024.png` is an additional full-size source DiT reference with the **same INT4 text conditioning and initial noise** as the native run. A ComfyUI seed uses a different RNG implementation and need not produce the same image as MLX. INT4 and tiled VAE decoding also prevent pixel-exact equivalence. ### 5. Schedule and regression tests `Packages/bonsai-runtime/Tests/BonsaiRuntimeTests/Fixtures/comfy-klein-simple.json` contains numerical output from unmodified ComfyUI functions at revision `65787d668397d230bf5839d69a0a7239e2dad378`. To regenerate, download the external `comfy/samplers.py` and `comfy/model_sampling.py` from that exact revision into a scratch reference directory, then run: ```sh "$KLEIN_WORK/env/bin/python" Tools/FluxKlein/export_schedule_fixture.py \ --references "$KLEIN_WORK/comfy-reference" \ --revision 65787d668397d230bf5839d69a0a7239e2dad378 \ --output "$KLEIN_WORK/comfy-klein-simple.json" ( cd Packages/bonsai-runtime MTL_DEBUG_LAYER=1 xcodebuild test -scheme bonsai-runtime-Package \ -destination 'platform=macOS' -parallel-testing-enabled NO \ -derivedDataPath "$KLEIN_WORK/runtime-build" \ -clonedSourcePackagesDirPath "$KLEIN_WORK/SourcePackages" \ -resultBundlePath "$KLEIN_WORK/package-tests.xcresult" ) xcodebuild build-for-testing -project LocalMuse.xcodeproj -scheme LocalMuse \ -destination 'generic/platform=iOS' -parallel-testing-enabled NO xcodebuild test -project LocalMuse.xcodeproj -scheme LocalMuse \ -destination 'platform=iOS Simulator,id=E623201E-7B34-4622-9EF5-CD4E43558F26' \ -parallel-testing-enabled NO -only-testing:LocalMuseTests ``` Reuse an existing simulator on the newest installed compatible runtime; substitute its UUID if the listed device is unavailable. The final application run executed 490 Swift Testing tests: 488 passed and the two physical-device opt-in tests were skipped. The full scheme's XCUITests also ran with Klein in the catalog; their only failure, `testModalGalleryFitsContentInBothAppearances`, is a pre-existing modal-height check that uses no Klein code. The package run passed all 18 tests with Metal API validation enabled. A clean rerun of the native Text to Image render after the encoder work produced a byte-identical `native-1024.png` with the same 3,606,440,960-byte MLX peak. Saved test and build summaries are in `validation/`; physical-device tests require an explicit opt-in and are skipped in normal regression runs. ## Publication and catalog installation The source was authenticated, converted and validated before publication. The six original runtime files were published at `c2abb5051572bcda02841b3e24af3e3fe9911d57`; `vae_encoder.safetensors` and the seven-file manifest were added at `646b51828f39b2541fbfcc643d6984d199c6f6a5` without changing those six files. Every remote byte size and LFS SHA-256 was compared with the local manifest. Small files were downloaded and hashed; each pinned public URL was also checked without authorization. Full evidence: `verification.json`. When replacing an artifact or publishing a new checkpoint: 1. Validate its source license, actual architecture and complete checkpoint hash. 2. Convert in a new directory and finish native/reference checks. 3. Authenticate to HF using `hf auth login`; keep credentials out of scripts/logs. 4. Create a private repository, upload the runtime files plus model card, license, notice, source metadata and manifest with `huggingface_hub.HfApi.upload_folder`. 5. Verify all remote sizes/hashes against the local manifest, publish it, and verify anonymous access to the resulting immutable commit. 6. Put **that commit** and every exact size and SHA-256 hash in the catalog. Never use `main` in an application download manifest. 7. Add the preview asset **and** the allowlist entry in `ModelIdentityIcon`. Match metadata in `PreviewCatalog` and add profile/catalog/lifecycle tests. Current catalog version is **60**; the 24th image model uses this manifest and profile. The default image model and language catalog remain unchanged. Model selection is lazy and all generation shares `MLExecutionGate`. Cancellation keeps synchronous MLX work registered until it unwinds. The runtime checks cancellation and the foreground gate before each encoder layer, transformer block and VAE operation. The 64 MiB MLX cache limit is restored at exit. Preparation reports real stages: text encoder and image transformer for Text to Image, plus the image encoder between them for Image to Image. Safetensors load lazily, so the transformer weights are evaluated inside their stage. The VAE decoder is loaded only at decoding after transformer eviction. Optional CPU clean-latent previews use the existing request-scoped relay and filtering limits (128 pixels, or 12 protected pixels), without a new neural network or changes to final latents. ## Validation and manual iPhone check Native Swift **1024 × 1024 / four-step** inference on an **M1 Pro, 16 GB Mac**: - Elapsed: **99.47 seconds**, with Metal API validation enabled. - Peak MLX allocation: **3,606,375,424 bytes** (3439.3 MiB / 3.36 GiB). - Preparation counts: `[0, 1, 2]`; four live preview frames, 128 × 128. - Native image: `validation/native-1024.png`; parameters and preview saved beside it. Reference editing at **1024 × 1024 / four steps** on the same Mac (`validation/native-1024-edit.png`, source `native-1024.png`): - Elapsed: **173.81 seconds**, with Metal API validation enabled. - Peak MLX allocation: **4,349,946,880 bytes** (4148.4 MiB / 4.05 GiB); the 4096 reference tokens lengthen every transformer step. - Preparation counts: `[0, 1, 2, 3]`; four live preview frames, 128 × 128. These are Mac measurements, **not iPhone speed or total-process-memory claims**. The physical-device check is left to the user. The current 12 GiB catalog floor is conservative; do not enable 8 GB devices based on host allocations alone. For normal manual testing, build/run LocalMuse, download **FLUX.2 Klein 4B**, keep the default 1024 square / four steps / CFG 1, and use the neutral English prompt above. Check both model-preparation stages, four denoising previews, correct final dimensions and image quality. Then open Image to Image with that result as the source and the edit prompt above: there is no strength control, the prompt is required, preparation shows three stages (text encoder, image encoder, image transformer) and only the requested change should differ. Also check cancellation, returning from the background, and switching back to an existing SD/SDXL or Z-Image model. A developer-only `KleinDeviceTests.installedCheckpointGeneratesAndEditsNativeCanvasWithMetalValidation()` smoke test is available with `LOCALMUSE_KLEIN_DEVICE_QA=1` and `MTL_DEBUG_LAYER=1`. It renders the prompt above, then reference-edits that render (`klein-device-qa` and `klein-device-qa-edit` PNG/preview/JSON in Caches, including peak MLX bytes). It uses the installed authenticated model or an explicitly staged cache directory and does not write gallery or persistence records. It is not enabled or run automatically. ## Cleanup Preserve source checkpoints in Downloads and these repository files. After publication and evidence checks, remove only the task's scratch environment, converted duplicate, external references, validation arrays and build products. Do not delete installed app models, user gallery data, shared HF caches or the app's existing SourcePackages cache. This run's temporary package `.build` directory and generated `default.profraw` can also be removed after validation.