Haru-4B GGUF (ROCmFPX)

GGUF files of higurashi-ai/Haru-4B, a full-duplex spoken dialogue model by higurashi-ai, with a native C++ runtime (llama-haru) in the ROCmFPX fork of llama.cpp.

Haru listens and speaks at the same time. Every 133 ms it takes one frame of your audio and produces one frame of its own audio and text. The language model is a Qwen3.5-4B text backbone. A small MeanFlow head turns its hidden state into a 64-dim audio latent, and a causal VibeVoice codec turns that into 24 kHz audio. All credit for the model goes to higurashi-ai (and to the Qwen team for the backbone). See the original card and the Haru code.

Files

File Size What it is
Haru-4B-Q8_0_ROCMFPX.gguf 4.37 GB Language model, ROCmFPX 8-bit with tensor scales (needs the fork). Recommended
Haru-4B-Q6_0_ROCMFPX.gguf 3.48 GB Language model, ROCmFPX 6-bit with tensor scales (needs the fork)
Haru-4B-audio-F16.gguf 2.30 GB Audio parts: the two audio projectors, the MeanFlow head and the codec (needed with either model)

You need one language model file and the audio file.

Run it

Build the fork, then run an offline conversation from two WAV files (24 kHz mono):

git clone https://github.com/HarshaVardhan3002/ROCmFPX && cd ROCmFPX
cmake -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build --target llama-haru -j

./build/bin/llama-haru -m Haru-4B-Q8_0_ROCMFPX.gguf --audio Haru-4B-audio-F16.gguf \
    --voice my_voice.wav --user question.wav --card-file card.txt -o answer.wav
  • --voice: the voice Haru speaks with. One speaker, 3 seconds to 3 minutes, clean speech.
  • --user: what you say. Haru listens to it frame by frame and answers while it plays.
  • --card / --card-file: the character card (system prompt). The Haru README shows the card layout the model was trained with.
  • Output: answer.wav (Haru's channel) and answer-stereo.wav (you on the left, Haru on the right). Haru's text appears on the terminal as it speaks.
  • Sampling defaults follow the official code: temperature 0.7, top-k 25, guidance 1.5, DRY on. --speak-first, --seed, --temp, --guidance and --tail (seconds of listening after your clip) are available.

The runtime is an offline (file in, file out) tool for now. A live microphone mode is not part of this release.

Accuracy

The runtime was checked part by part against the official PyTorch code in float32, on a real voice clip and 24 live dialogue frames:

Part Cosine similarity to PyTorch
Codec encode (whole clip and streaming), streaming decode 1.000000
Audio projectors (input embeddings) 1.000000
Language model hidden states 0.999998
Text logits 0.999999 (same top token on 24 / 24 frames)
MeanFlow head latents 1.000000

Quantized language models, KL divergence to BF16 on wiki text (40 x 512 tokens), and hidden-state similarity on the real dialogue frames (file run as is on the CPU):

Language model Size Wiki KLD Dialogue hidden cosine
Q8_0_ROCMFPX (this repo) 4.37 GB 0.0019 0.9966
stock Q8_0 (not uploaded) 4.48 GB 0.0031 0.9954
Q6_0_ROCMFPX (this repo) 3.48 GB 0.0113 0.9915
stock Q6_K (not uploaded) 3.46 GB 0.0153 0.9913

On the CPU, use the fork at commit 4310e2480 or newer for the ROCmFPX 8-bit file. Older builds rounded the activations to 8 bits before each matrix multiply, which brought this file down to 0.966 on the dialogue frames. The audio file in F16 matches the float32 one (head cosine 0.9999997).

Notes and limits

  • Verified on the CPU (ARM). On a CPU it is slower than real time; real-time use needs a GPU build. GPU backends have not been tested with this runtime yet.
  • Haru-4B is a pretrained base model released for research. It holds a natural conversation, but it is not a reliable source of facts and does not follow instructions the way an assistant does.
  • It can imitate the voice of a reference clip. Only use voices you have the right to use.

License

Apache License 2.0, same as the original weights.

Downloads last month
-
GGUF
Model size
1B params
Architecture
haru-audio
Hardware compatibility
Log In to add your hardware

8-bit

16-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for Harsha001/Haru-4B-ROCmFPX

Quantized
(6)
this model