Commit ·
0c8dd89
0
Parent(s):
Duplicate from higurashi-ai/Haru-4B
Browse filesCo-authored-by: Kotoge <Kotoge@users.noreply.huggingface.co>
- .gitattributes +36 -0
- README.md +110 -0
- assets/architecture.svg +23 -0
- config.json +131 -0
- model.safetensors +3 -0
- tokenizer.json +3 -0
- tokenizer_config.json +14 -0
.gitattributes
ADDED
|
@@ -0,0 +1,36 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
*.7z filter=lfs diff=lfs merge=lfs -text
|
| 2 |
+
*.arrow filter=lfs diff=lfs merge=lfs -text
|
| 3 |
+
*.bin filter=lfs diff=lfs merge=lfs -text
|
| 4 |
+
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
| 5 |
+
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
| 6 |
+
*.ftz filter=lfs diff=lfs merge=lfs -text
|
| 7 |
+
*.gz filter=lfs diff=lfs merge=lfs -text
|
| 8 |
+
*.h5 filter=lfs diff=lfs merge=lfs -text
|
| 9 |
+
*.joblib filter=lfs diff=lfs merge=lfs -text
|
| 10 |
+
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
| 11 |
+
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
| 12 |
+
*.model filter=lfs diff=lfs merge=lfs -text
|
| 13 |
+
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
| 14 |
+
*.npy filter=lfs diff=lfs merge=lfs -text
|
| 15 |
+
*.npz filter=lfs diff=lfs merge=lfs -text
|
| 16 |
+
*.onnx filter=lfs diff=lfs merge=lfs -text
|
| 17 |
+
*.ot filter=lfs diff=lfs merge=lfs -text
|
| 18 |
+
*.parquet filter=lfs diff=lfs merge=lfs -text
|
| 19 |
+
*.pb filter=lfs diff=lfs merge=lfs -text
|
| 20 |
+
*.pickle filter=lfs diff=lfs merge=lfs -text
|
| 21 |
+
*.pkl filter=lfs diff=lfs merge=lfs -text
|
| 22 |
+
*.pt filter=lfs diff=lfs merge=lfs -text
|
| 23 |
+
*.pth filter=lfs diff=lfs merge=lfs -text
|
| 24 |
+
*.rar filter=lfs diff=lfs merge=lfs -text
|
| 25 |
+
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
| 26 |
+
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
| 27 |
+
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
| 28 |
+
*.tar filter=lfs diff=lfs merge=lfs -text
|
| 29 |
+
*.tflite filter=lfs diff=lfs merge=lfs -text
|
| 30 |
+
*.tgz filter=lfs diff=lfs merge=lfs -text
|
| 31 |
+
*.wasm filter=lfs diff=lfs merge=lfs -text
|
| 32 |
+
*.xz filter=lfs diff=lfs merge=lfs -text
|
| 33 |
+
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
+
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
+
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
README.md
ADDED
|
@@ -0,0 +1,110 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
language:
|
| 4 |
+
- en
|
| 5 |
+
- es
|
| 6 |
+
- pt
|
| 7 |
+
- de
|
| 8 |
+
- fr
|
| 9 |
+
- zh
|
| 10 |
+
- ja
|
| 11 |
+
- ru
|
| 12 |
+
- ko
|
| 13 |
+
pipeline_tag: audio-to-audio
|
| 14 |
+
tags:
|
| 15 |
+
- speech
|
| 16 |
+
- full-duplex
|
| 17 |
+
- speech-to-speech
|
| 18 |
+
- haru
|
| 19 |
+
---
|
| 20 |
+
|
| 21 |
+
# Haru-4B
|
| 22 |
+
|
| 23 |
+
[Code](https://github.com/higurashi-ai/Haru) ·
|
| 24 |
+
[Technical blog](https://blog.higurashi.ai/melte/haru4b/) ·
|
| 25 |
+
[Technical blog (Japanese)](https://blog.higurashi.ai/melte/haru4b-ja/)
|
| 26 |
+
|
| 27 |
+
Haru-4B is a low-latency and multilingual full-duplex continuous spoken dialogue foundation
|
| 28 |
+
model: it listens and speaks at the same time, the way people do on a phone call. It is a
|
| 29 |
+
pretrained base model, released for research.
|
| 30 |
+
|
| 31 |
+
- **Native full-duplex at 7.5 Hz.** Every 133 ms the model takes one frame of the user's audio
|
| 32 |
+
and produces one frame of its own audio and text. User audio, model audio and model text are
|
| 33 |
+
summed into a single position of the language model, so listening and speaking never wait
|
| 34 |
+
for each other.
|
| 35 |
+
- **Continuous audio, one sampling step.** Audio is a 64-dimensional latent per frame. A small
|
| 36 |
+
MLP head with MeanFlow sampling produces it in one step from the language model's hidden
|
| 37 |
+
state; there is no separate audio transformer and there are no discrete audio tokens.
|
| 38 |
+
- **Character and voice control.** A text card sets the character, a short reference clip sets
|
| 39 |
+
the voice.
|
| 40 |
+
- **Multilingual.** English, Spanish, Portuguese, German, French, Chinese, Japanese, Russian
|
| 41 |
+
and Korean.
|
| 42 |
+
|
| 43 |
+
<p align="center">
|
| 44 |
+
<img src="assets/architecture.svg" width="720" alt="Haru-4B architecture: in every 133 ms frame, the user's speech latent, the model's previous text token and its previous speech latent are summed into one position of the language model, whose output predicts the next text token and the next speech latent.">
|
| 45 |
+
</p>
|
| 46 |
+
|
| 47 |
+
## Usage
|
| 48 |
+
|
| 49 |
+
The code is at <https://github.com/higurashi-ai/Haru>: a PyTorch implementation with a
|
| 50 |
+
web demo, an MLX implementation with a native macOS app, and a plugin for serving with
|
| 51 |
+
vLLM-Omni.
|
| 52 |
+
|
| 53 |
+
With an NVIDIA GPU (about 12 GB of memory):
|
| 54 |
+
|
| 55 |
+
```bash
|
| 56 |
+
git clone https://github.com/higurashi-ai/Haru && cd Haru
|
| 57 |
+
pip install -e ./haru
|
| 58 |
+
python -m haru.server --hf-repo higurashi-ai/Haru-4B
|
| 59 |
+
```
|
| 60 |
+
|
| 61 |
+
Open <http://localhost:8998>, pick a character, give the model a voice, and talk. For low
|
| 62 |
+
latency, we recommend wired headphones and a wired microphone. Bluetooth devices add 100 ms or
|
| 63 |
+
more.
|
| 64 |
+
|
| 65 |
+
**Bring your own voice.** Haru ships without voice clips. Upload or record a clip of the voice
|
| 66 |
+
the model should speak with: one speaker, 3 seconds to 3 minutes, clean speech. Only use voices
|
| 67 |
+
you have the right to use.
|
| 68 |
+
|
| 69 |
+
```python
|
| 70 |
+
from haru import DuplexSession, HaruModel
|
| 71 |
+
from haru.audio import load_audio, prepare_reference
|
| 72 |
+
|
| 73 |
+
model = HaruModel.from_pretrained("higurashi-ai/Haru-4B")
|
| 74 |
+
voice = prepare_reference(load_audio("voice.wav"))
|
| 75 |
+
session = DuplexSession(model, system_prompt=card, reference=voice)
|
| 76 |
+
|
| 77 |
+
frame = session.start() # the model's first 133 ms: frame.audio, frame.text
|
| 78 |
+
while not frame.done:
|
| 79 |
+
frame = session.step(user_audio) # 3200 samples at 24 kHz in, 3200 out
|
| 80 |
+
```
|
| 81 |
+
|
| 82 |
+
The system prompt is a character card with a fixed layout; the repository's README shows how
|
| 83 |
+
to build one.
|
| 84 |
+
|
| 85 |
+
## Files
|
| 86 |
+
|
| 87 |
+
| File | Contents |
|
| 88 |
+
| --- | --- |
|
| 89 |
+
| `model.safetensors` | All weights (bfloat16): `lm.*` language model, `model_audio_proj.*` / `user_audio_proj.*` audio input projections, `head.*` MeanFlow head, `codec.*` audio codec |
|
| 90 |
+
| `config.json` | Model configuration |
|
| 91 |
+
| `tokenizer.json` | Text tokenizer |
|
| 92 |
+
|
| 93 |
+
## Architecture
|
| 94 |
+
|
| 95 |
+
| Part | Details |
|
| 96 |
+
| --- | --- |
|
| 97 |
+
| Language model | Qwen3.5-4B text backbone, 32 layers, hidden size 2560 |
|
| 98 |
+
| Audio codec | VibeVoice acoustic tokenizer, causal, 24 kHz, 3200 samples per latent |
|
| 99 |
+
| Audio head | 4-layer MLP, MeanFlow, 1 sampling step, guidance scale 1 to 2 (default 1.5) |
|
| 100 |
+
| Frame rate | 7.5 Hz |
|
| 101 |
+
|
| 102 |
+
## Limitations
|
| 103 |
+
|
| 104 |
+
Haru-4B is a pretrained base model, released for research. It holds a natural conversation,
|
| 105 |
+
but it is not a reliable source of facts and does not follow instructions the way an assistant
|
| 106 |
+
does. It can imitate the voice of a reference clip; only use voices you have the right to use.
|
| 107 |
+
|
| 108 |
+
## License
|
| 109 |
+
|
| 110 |
+
The weights are released under the Apache License 2.0.
|
assets/architecture.svg
ADDED
|
|
config.json
ADDED
|
@@ -0,0 +1,131 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"model_type": "haru",
|
| 3 |
+
"architectures": [
|
| 4 |
+
"HaruForConditionalGeneration"
|
| 5 |
+
],
|
| 6 |
+
"dtype": "bfloat16",
|
| 7 |
+
"sample_rate": 24000,
|
| 8 |
+
"frame_samples": 3200,
|
| 9 |
+
"frame_rate": 7.5,
|
| 10 |
+
"latent_dim": 64,
|
| 11 |
+
"pad_token_id": 248076,
|
| 12 |
+
"eop_token_id": 248071,
|
| 13 |
+
"lm": {
|
| 14 |
+
"model_type": "qwen3_5_text",
|
| 15 |
+
"vocab_size": 248320,
|
| 16 |
+
"hidden_size": 2560,
|
| 17 |
+
"intermediate_size": 9216,
|
| 18 |
+
"num_hidden_layers": 32,
|
| 19 |
+
"num_attention_heads": 16,
|
| 20 |
+
"num_key_value_heads": 4,
|
| 21 |
+
"head_dim": 256,
|
| 22 |
+
"hidden_act": "silu",
|
| 23 |
+
"max_position_embeddings": 262144,
|
| 24 |
+
"rms_norm_eps": 1e-06,
|
| 25 |
+
"attention_bias": false,
|
| 26 |
+
"attention_dropout": 0.0,
|
| 27 |
+
"attn_output_gate": true,
|
| 28 |
+
"full_attention_interval": 4,
|
| 29 |
+
"layer_types": [
|
| 30 |
+
"linear_attention",
|
| 31 |
+
"linear_attention",
|
| 32 |
+
"linear_attention",
|
| 33 |
+
"full_attention",
|
| 34 |
+
"linear_attention",
|
| 35 |
+
"linear_attention",
|
| 36 |
+
"linear_attention",
|
| 37 |
+
"full_attention",
|
| 38 |
+
"linear_attention",
|
| 39 |
+
"linear_attention",
|
| 40 |
+
"linear_attention",
|
| 41 |
+
"full_attention",
|
| 42 |
+
"linear_attention",
|
| 43 |
+
"linear_attention",
|
| 44 |
+
"linear_attention",
|
| 45 |
+
"full_attention",
|
| 46 |
+
"linear_attention",
|
| 47 |
+
"linear_attention",
|
| 48 |
+
"linear_attention",
|
| 49 |
+
"full_attention",
|
| 50 |
+
"linear_attention",
|
| 51 |
+
"linear_attention",
|
| 52 |
+
"linear_attention",
|
| 53 |
+
"full_attention",
|
| 54 |
+
"linear_attention",
|
| 55 |
+
"linear_attention",
|
| 56 |
+
"linear_attention",
|
| 57 |
+
"full_attention",
|
| 58 |
+
"linear_attention",
|
| 59 |
+
"linear_attention",
|
| 60 |
+
"linear_attention",
|
| 61 |
+
"full_attention"
|
| 62 |
+
],
|
| 63 |
+
"linear_conv_kernel_dim": 4,
|
| 64 |
+
"linear_key_head_dim": 128,
|
| 65 |
+
"linear_value_head_dim": 128,
|
| 66 |
+
"linear_num_key_heads": 16,
|
| 67 |
+
"linear_num_value_heads": 32,
|
| 68 |
+
"mamba_ssm_dtype": "float32",
|
| 69 |
+
"partial_rotary_factor": 0.25,
|
| 70 |
+
"rope_parameters": {
|
| 71 |
+
"mrope_interleaved": true,
|
| 72 |
+
"mrope_section": [
|
| 73 |
+
11,
|
| 74 |
+
11,
|
| 75 |
+
10
|
| 76 |
+
],
|
| 77 |
+
"partial_rotary_factor": 0.25,
|
| 78 |
+
"rope_theta": 10000000,
|
| 79 |
+
"rope_type": "default"
|
| 80 |
+
},
|
| 81 |
+
"tie_word_embeddings": true,
|
| 82 |
+
"initializer_range": 0.02,
|
| 83 |
+
"eos_token_id": 248044
|
| 84 |
+
},
|
| 85 |
+
"codec": {
|
| 86 |
+
"latent_dim": 64,
|
| 87 |
+
"base_channels": 32,
|
| 88 |
+
"ratios": [
|
| 89 |
+
8,
|
| 90 |
+
5,
|
| 91 |
+
5,
|
| 92 |
+
4,
|
| 93 |
+
2,
|
| 94 |
+
2
|
| 95 |
+
],
|
| 96 |
+
"encoder_depths": [
|
| 97 |
+
3,
|
| 98 |
+
3,
|
| 99 |
+
3,
|
| 100 |
+
3,
|
| 101 |
+
3,
|
| 102 |
+
3,
|
| 103 |
+
8
|
| 104 |
+
],
|
| 105 |
+
"decoder_depths": [
|
| 106 |
+
8,
|
| 107 |
+
3,
|
| 108 |
+
3,
|
| 109 |
+
3,
|
| 110 |
+
3,
|
| 111 |
+
3,
|
| 112 |
+
3
|
| 113 |
+
],
|
| 114 |
+
"norm_eps": 1e-05,
|
| 115 |
+
"latent_std": 0.5
|
| 116 |
+
},
|
| 117 |
+
"head": {
|
| 118 |
+
"hidden_size": 2560,
|
| 119 |
+
"latent_dim": 64,
|
| 120 |
+
"num_layers": 4,
|
| 121 |
+
"ffn_ratio": 3.0,
|
| 122 |
+
"norm_eps": 1e-05,
|
| 123 |
+
"interval_embed": 7.0,
|
| 124 |
+
"num_steps": 1,
|
| 125 |
+
"guidance_scale": 1.5,
|
| 126 |
+
"guidance_range": [
|
| 127 |
+
1.0,
|
| 128 |
+
2.0
|
| 129 |
+
]
|
| 130 |
+
}
|
| 131 |
+
}
|
model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:ad2c3534ef6bd88525e22e5da3ac4fb739368406bf3b1c852cbafbade647c0fa
|
| 3 |
+
size 10540113510
|
tokenizer.json
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:06b9509352d2af50381ab2247e083b80d32d5c0aba91c272ca9ff729b6a0e523
|
| 3 |
+
size 19989325
|
tokenizer_config.json
ADDED
|
@@ -0,0 +1,14 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"add_prefix_space": false,
|
| 3 |
+
"backend": "tokenizers",
|
| 4 |
+
"bos_token": null,
|
| 5 |
+
"clean_up_tokenization_spaces": false,
|
| 6 |
+
"eos_token": "<|im_end|>",
|
| 7 |
+
"errors": "replace",
|
| 8 |
+
"model_max_length": 262144,
|
| 9 |
+
"pad_token": "<|endoftext|>",
|
| 10 |
+
"pretokenize_regex": "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?[\\p{L}\\p{M}]+|\\p{N}| ?[^\\s\\p{L}\\p{M}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+",
|
| 11 |
+
"split_special_tokens": false,
|
| 12 |
+
"tokenizer_class": "Qwen2Tokenizer",
|
| 13 |
+
"unk_token": null
|
| 14 |
+
}
|