Haru-4B MLX 8-bit

An 8-bit MLX quantization of higurashi-ai/Haru-4B, the full-duplex spoken dialogue model from Higurashi, for Apple silicon. It is 6.6 GB instead of 10.5 GB and about 1.5x faster per frame than bf16 (34 vs 50 ms on an M4 Pro), and follows the bf16 model closely (same top text token on 97.6% of frames, see Benchmark). Its output matches upstream's own on-the-fly --quantize 8 mode.

This is an unofficial community conversion. All credit for the model goes to the Haru authors: see the original model card, code and technical blog.

What was quantized

The quantization is the same as upstream's HaruModel.quantize(bits=8), saved to disk so you don't have to download the bf16 checkpoint and quantize it at every load:

Part Precision
Qwen3.5 backbone weight matrices (32 layers) 8-bit affine, group size 64
Token embedding table (tied output head) 8-bit affine, group size 64
Norm gains, Gated DeltaNet conv and gates, audio projectors checkpoint precision (bf16 / fp32)
MeanFlow audio head checkpoint precision
VibeVoice audio codec checkpoint precision

model.safetensors holds the model's parameters after conversion to MLX layout and quantization, so it is self-contained (backbone, head and codec), but it is not in the original Hugging Face key layout. Upstream's haru_mlx.models.load_model, and the native macOS app, can't read it. Use the load_haru_8bit.py loader in this repo.

4-bit is not offered. In our test it collapsed into mixed-script nonsense, which matches upstream's note that at 4 bits the hidden states no longer follow the float model.

Benchmark vs the bf16 original

Run on an M4 Pro (24 GB) with haru_mlx (offline wav-to-wav, the same frame-by-frame loop as a live call). Inputs: 6 English opening lines (macOS TTS, padded to 20 s = 150 frames), an English character card and a 23 s reference voice. Default sampling (temperature 0.7, top-k 25, guidance 1.5).

Speed and memory

18 conversations per precision (6 prompts x 3 seeds, 2,682 frames each). The realtime budget is 133 ms per frame (7.5 Hz).

Precision ms / frame (median) p95 Realtime factor Peak MLX memory Load time Weights on disk
bf16 (original) 50.4 51.4 0.38 11.6 GB 5.7 s 10.5 GB
8-bit (this repo) 34.1 35.0 0.26 7.7 GB 1.3 s 6.6 GB

Upstream's on-the-fly --quantize 8 produces the same model but has to read the 10.5 GB bf16 checkpoint and quantize it at every load (3.7 s here). Load times were measured with the files already in the OS cache, so a cold first load takes longer.

Fidelity

How closely the 8-bit model follows bf16. For each prompt the 8-bit model is forced along the exact conversation bf16 produced (same text tokens, same audio latents, same sampling noise), so both models predict every frame from identical inputs. 6 conversations, 900 frames. Higher is better for agreement and cosine; lower is better for KL and error.

vs bf16 Text top-1 agreement Text KL (mean) Text KL (p99) Audio latent cosine Audio latent rel. error
8-bit (this repo) 97.6% 0.004 0.085 0.992 0.08
4-bit (reference, not released) 46.9% 3.03 17.9 0.706 0.76

The 8-bit model picks the same most-likely text token as bf16 on 97.6% of frames. The worst prompt was 96.0% and the best was 100%. 4-bit is included only to show what broken quantization looks like: its free-running output is mixed-script nonsense.

End-to-end replies

The same 18 free-running conversations per precision, with the model's audio transcribed by Whisper (faster-whisper small.en). "Text-vs-audio WER" compares the words the model wrote in its text stream with what Whisper hears in its audio, so it measures whether the speech is intelligible and matches the text.

Precision Replies with 3+ words Words per reply Text-vs-audio WER
bf16 (original) 16 / 18 22.2 21.3%
8-bit (this repo) 16 / 18 28.2 14.6%

Both precisions give coherent, on-topic replies. The short replies ("Yeah,", one silent run) occur at both precisions and are base-model behavior. The WER is inflated for both by name mishearings ("Kavin" heard as "Kevin") and numbers ("22" vs "twenty two"). The gap between the two is within the noise of 18 samples, so read it as "no measurable loss", not "8-bit is better".

Caveats: one machine, English only, synthetic (TTS) inputs and voice, small sample. A Mac with 16 GB of RAM should fit the 7.7 GB peak, but only the M4 Pro was tested.

Usage

Install the MLX implementation of Haru (Python 3.10+, Apple silicon):

pip install "git+https://github.com/higurashi-ai/Haru#subdirectory=haru_mlx"
hf download nano16/Haru-4B-mlx-8bit --local-dir Haru-4B-mlx-8bit
cd Haru-4B-mlx-8bit

Offline wav-to-wav. Everything after the model directory is a normal haru-inference flag:

python load_haru_8bit.py . user.wav reply.wav \
  --reference voice.wav --system-prompt-file card.txt --stereo

In Python, load() returns the same HaruModel that haru_mlx.models.load_model returns:

from load_haru_8bit import load
from haru_mlx.session import DuplexSession

model = load(".")
# continue exactly as in the haru_mlx README: DuplexSession(model, system_prompt=card, reference=voice)

See the Haru README to write a character card with render_card. Use wired headphones for live use; Bluetooth adds 100 ms or more.

Tested with mlx 0.32.4 and haru_mlx from upstream commit 7fe6b76.

Limitations

Everything in the original model card applies:

  • Haru-4B is a pretrained base model released for research. It holds a natural conversation, but it is not a reliable source of facts and does not follow instructions like an assistant.
  • No voices ship with it. The model imitates the voice of the reference clip you give it. Only use voices you have the right to use, and don't use it to impersonate real people.
  • Quantization can add small errors on top of the base model's. Check output for your use case.

License

Apache License 2.0, same as the original weights and code (see LICENSE). Haru-4B starts from Qwen3.5-4B and includes third-party work such as the VibeVoice acoustic tokenizer. See THIRD_PARTY_NOTICES.md, copied unchanged from the upstream repository.

Downloads last month
61
Safetensors
Model size
2B params
Tensor type
BF16
路
U32
路
F32
路
MLX
Hardware compatibility
Log In to add your hardware

8-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. 馃檵 Ask for provider support

Model tree for nano16/Haru-4B-mlx-8bit

Quantized
(6)
this model