devin-lai's picture
Prepare inference-only v0.5.0 runtime cleanup
63e9703 verified
|
Raw History Blame
6.41 kB

Installation and inference

Use an Apple silicon Mac, native arm64 Python 3.12 and MLX 0.32.4. For a source install, follow the quick start. The release can also be installed directly from Git:

uv pip install 'minimax-h3-image-mlx[native] @ git+https://github.com/devin-lai/MiniMax-H3-Image-MLX.git@v0.5.0'

.[native] supplies the CPU PyTorch RNG used upstream; the neural operators run with MLX. --rng mlx avoids PyTorch for text-to-image generation but uses a different noise stream. Editing needs the original PyTorch generator for posterior and anchor sampling. Use detailed English prompts; prompt expansion is not automatic.

Download once, infer locally

hf auth login
h3-mlx download --variant packed12 --model-dir models/packed12

Request access to the variant and shared repositories first. The downloader uses immutable revisions from model_index.json, fetches only the selected variant and links reusable components from the Hub cache. Use a separate assembly directory for each variant/release.

h3-mlx download --variant q8 --model-dir models/q8 \
  --text-encoder-path /path/to/existing/text_encoder
h3-mlx download --variant q4 --model-dir models/q4 \
  --skip-text-encoder --verify

--verify rehashes all component files against their SHA-256 manifests. --local-files-only uses an already populated Hub cache without network access. The assembly uses symlinks: keep the Hub cache available, or dereference those links when moving weights to another machine. No model files are bundled with the Python package. Inference never downloads remote Python or enables trust_remote_code.

h3-mlx generate --model models/packed12 \
  --text-encoder models/packed12/text_encoder \
  'A detailed editorial photograph of a blue glass bottle on pale stone' \
  --seed 731 --width 2048 --height 2048 --output outputs/bottle.png

Each PNG is accompanied by a JSON record. Add --save-trace to retain the conditioning, initial state, six velocities, six Euler states and final latent for numerical comparisons. Traces consume additional disk space.

Reuse a resident pipeline

A new CLI process reloads weights. To reuse a model across requests:

from h3_mlx.runtime import Pipeline

pipe = Pipeline(
    "models/packed12",
    "models/packed12/text_encoder",
    compile_blocks=True,
    eval_every=0,
    prefetch_blocks=4,
    vae_compile=True,
    vae_batch=4,
    kernel="metal-qk-exact",
    tune_matmul=True,
)
try:
    for seed in (731, 732, 733):
        image, record = pipe.generate(
            "An editorial photograph of a cobalt perfume bottle on pale stone",
            seed=seed,
            output=f"outputs/bottle-{seed}.png",
        )
finally:
    pipe.clear()

Conditioning is cached by weights, token presentation, images and precision settings. Different seeds can reuse it. Refined conditioning is also cached exactly; --no-refiner-cache disables that second cache.

Measured exact tuning

The historical M5 Pro campaign used these CLI options:

--compile-blocks --eval-every 0 --prefetch-blocks 4 \
--compile-vae --vae-batch 4 --kernel metal-qk-exact --tune-matmul

The tested 64 GiB machine also used --wired-gb 32 for packed12 or --wired-gb 42 for BF16. These are process-local MLX limits; the runtime checks Metal's recommended working set. They are not universal settings for other Macs. The exact Q/K and FC2 kernels restrict dispatch to the tested hardware, MLX version and shapes, retaining native fallbacks.

The released frozen modulation tables require MLX 0.32.4 and MLX_ENABLE_TF32=0. Import h3_mlx before MLX in embedded applications: the package sets that variable and MLX_CONV_WINOGRAD=0 unless explicitly overridden. Overrides are recorded, and incompatible table precision is rejected.

The downloader selects packed12/Q8/Q4 precision from release metadata. There are no conversion or online-quantization switches. To change precision, download another published variant.

Reference-image editing

h3-mlx generate --model models/packed12 \
  --text-encoder models/packed12/text_encoder \
  'Change the bottle in <Picture 1> to emerald glass while preserving the label' \
  --image outputs/bottle.png --seed 42 --output outputs/edit.png

Repeat --image for ordered references, addressed as <Picture 1>, <Picture 2>, etc. The implementation accepts up to sixteen inputs; the recorded editing cohort validates one. This does not establish broad multi-reference editing quality.

The native minimum generation area is 4,194,304 pixels. Smaller requested outputs use an enlarged generation canvas and are resized after decoding. For example, the recorded 1024² edit used a 2048² canvas. --min-generation-area is a development override and changes the workload.

Memory and optional precision tradeoffs

Option Effect
--dit-cache-gb N Budget for resident DiT blocks; default 38 GiB, capped against the device working set
--mlp-chunk N Bound the MLP activation chunk; zero uses the complete sequence
--attention-chunk N Query chunking; zero uses one memory-efficient SDPA call
--offload-after-denoise Release DiT before VAE decoding
--vae-dtype float32 Use a separate FP32 decoder precision/performance tradeoff
--experimental-steps 0,2,4 Explicitly skip heads; changes images and is outside the six-step benchmark

Q8/Q4 reduce core storage and change images. These knobs do not establish a minimum supported RAM capacity. JSON distinguishes MLX allocator counters from parent physical footprint; process lifetime high water includes earlier requests. Those overlapping counters must not be added.

Compare actual outputs

Install the audit extra, then compare matching prompt/seed/canvas outputs:

python -m h3_mlx.compare \
  --reference outputs/bf16.png --candidate outputs/q8.png \
  --label BF16 --label Q8 --output outputs/comparison

The comparison tool checks input metadata and writes full-frame grids, identical native-resolution crops, amplified RGB differences and scores with source hashes. RGB SSIM/PSNR measure similarity to a reference trajectory, not prompt adherence or overall aesthetic quality.

Benchmarks and reproduction · Technical notes