Instructions to use nano16/Haru-4B-mlx-8bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use nano16/Haru-4B-mlx-8bit with MLX:
# Download the model from the Hub pip install huggingface_hub[hf_xet] hf download nano16/Haru-4B-mlx-8bit --local-dir Haru-4B-mlx-8bit
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Atomic Chat
Haru-4B MLX 8-bit
An 8-bit MLX quantization of
higurashi-ai/Haru-4B, the full-duplex spoken
dialogue model from Higurashi, for Apple silicon. It is 6.6 GB instead of 10.5 GB and
about 1.5x faster per frame than bf16 (34 vs 50 ms on an M4 Pro), and follows the bf16
model closely (same top text token on 97.6% of frames, see Benchmark).
Its output matches upstream's own on-the-fly --quantize 8 mode.
This is an unofficial community conversion. All credit for the model goes to the Haru authors: see the original model card, code and technical blog.
What was quantized
The quantization is the same as upstream's HaruModel.quantize(bits=8), saved to disk so you
don't have to download the bf16 checkpoint and quantize it at every load:
| Part | Precision |
|---|---|
| Qwen3.5 backbone weight matrices (32 layers) | 8-bit affine, group size 64 |
| Token embedding table (tied output head) | 8-bit affine, group size 64 |
| Norm gains, Gated DeltaNet conv and gates, audio projectors | checkpoint precision (bf16 / fp32) |
| MeanFlow audio head | checkpoint precision |
| VibeVoice audio codec | checkpoint precision |
model.safetensors holds the model's parameters after conversion to MLX layout and
quantization, so it is self-contained (backbone, head and codec), but it is not in the
original Hugging Face key layout. Upstream's haru_mlx.models.load_model, and the native macOS
app, can't read it. Use the load_haru_8bit.py loader in this repo.
4-bit is not offered. In our test it collapsed into mixed-script nonsense, which matches upstream's note that at 4 bits the hidden states no longer follow the float model.
Benchmark vs the bf16 original
Run on an M4 Pro (24 GB) with haru_mlx (offline wav-to-wav, the same frame-by-frame loop as a
live call). Inputs: 6 English opening lines (macOS TTS, padded to 20 s = 150 frames), an English
character card and a 23 s reference voice. Default sampling (temperature 0.7, top-k 25,
guidance 1.5).
Speed and memory
18 conversations per precision (6 prompts x 3 seeds, 2,682 frames each). The realtime budget is 133 ms per frame (7.5 Hz).
| Precision | ms / frame (median) | p95 | Realtime factor | Peak MLX memory | Load time | Weights on disk |
|---|---|---|---|---|---|---|
| bf16 (original) | 50.4 | 51.4 | 0.38 | 11.6 GB | 5.7 s | 10.5 GB |
| 8-bit (this repo) | 34.1 | 35.0 | 0.26 | 7.7 GB | 1.3 s | 6.6 GB |
Upstream's on-the-fly --quantize 8 produces the same model but has to read the 10.5 GB bf16
checkpoint and quantize it at every load (3.7 s here). Load times were measured with the files
already in the OS cache, so a cold first load takes longer.
Fidelity
How closely the 8-bit model follows bf16. For each prompt the 8-bit model is forced along the exact conversation bf16 produced (same text tokens, same audio latents, same sampling noise), so both models predict every frame from identical inputs. 6 conversations, 900 frames. Higher is better for agreement and cosine; lower is better for KL and error.
| vs bf16 | Text top-1 agreement | Text KL (mean) | Text KL (p99) | Audio latent cosine | Audio latent rel. error |
|---|---|---|---|---|---|
| 8-bit (this repo) | 97.6% | 0.004 | 0.085 | 0.992 | 0.08 |
| 4-bit (reference, not released) | 46.9% | 3.03 | 17.9 | 0.706 | 0.76 |
The 8-bit model picks the same most-likely text token as bf16 on 97.6% of frames. The worst prompt was 96.0% and the best was 100%. 4-bit is included only to show what broken quantization looks like: its free-running output is mixed-script nonsense.
End-to-end replies
The same 18 free-running conversations per precision, with the model's audio transcribed by Whisper (faster-whisper small.en). "Text-vs-audio WER" compares the words the model wrote in its text stream with what Whisper hears in its audio, so it measures whether the speech is intelligible and matches the text.
| Precision | Replies with 3+ words | Words per reply | Text-vs-audio WER |
|---|---|---|---|
| bf16 (original) | 16 / 18 | 22.2 | 21.3% |
| 8-bit (this repo) | 16 / 18 | 28.2 | 14.6% |
Both precisions give coherent, on-topic replies. The short replies ("Yeah,", one silent run) occur at both precisions and are base-model behavior. The WER is inflated for both by name mishearings ("Kavin" heard as "Kevin") and numbers ("22" vs "twenty two"). The gap between the two is within the noise of 18 samples, so read it as "no measurable loss", not "8-bit is better".
Caveats: one machine, English only, synthetic (TTS) inputs and voice, small sample. A Mac with 16 GB of RAM should fit the 7.7 GB peak, but only the M4 Pro was tested.
Usage
Install the MLX implementation of Haru (Python 3.10+, Apple silicon):
pip install "git+https://github.com/higurashi-ai/Haru#subdirectory=haru_mlx"
hf download nano16/Haru-4B-mlx-8bit --local-dir Haru-4B-mlx-8bit
cd Haru-4B-mlx-8bit
Offline wav-to-wav. Everything after the model directory is a normal haru-inference flag:
python load_haru_8bit.py . user.wav reply.wav \
--reference voice.wav --system-prompt-file card.txt --stereo
In Python, load() returns the same HaruModel that haru_mlx.models.load_model returns:
from load_haru_8bit import load
from haru_mlx.session import DuplexSession
model = load(".")
# continue exactly as in the haru_mlx README: DuplexSession(model, system_prompt=card, reference=voice)
See the Haru README to write a character
card with render_card. Use wired headphones for live use; Bluetooth adds 100 ms or more.
Tested with mlx 0.32.4 and haru_mlx from upstream commit 7fe6b76.
Limitations
Everything in the original model card applies:
- Haru-4B is a pretrained base model released for research. It holds a natural conversation, but it is not a reliable source of facts and does not follow instructions like an assistant.
- No voices ship with it. The model imitates the voice of the reference clip you give it. Only use voices you have the right to use, and don't use it to impersonate real people.
- Quantization can add small errors on top of the base model's. Check output for your use case.
License
Apache License 2.0, same as the original weights and code (see LICENSE). Haru-4B starts from
Qwen3.5-4B and includes third-party work such as the VibeVoice acoustic tokenizer. See
THIRD_PARTY_NOTICES.md, copied unchanged from the upstream repository.
- Downloads last month
- 61
8-bit
Model tree for nano16/Haru-4B-mlx-8bit
Base model
higurashi-ai/Haru-4B