hyperakrep Kotoge commited on
Commit
0c8dd89
·
0 Parent(s):

Duplicate from higurashi-ai/Haru-4B

Browse files

Co-authored-by: Kotoge <Kotoge@users.noreply.huggingface.co>

.gitattributes ADDED
@@ -0,0 +1,36 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ *.7z filter=lfs diff=lfs merge=lfs -text
2
+ *.arrow filter=lfs diff=lfs merge=lfs -text
3
+ *.bin filter=lfs diff=lfs merge=lfs -text
4
+ *.bz2 filter=lfs diff=lfs merge=lfs -text
5
+ *.ckpt filter=lfs diff=lfs merge=lfs -text
6
+ *.ftz filter=lfs diff=lfs merge=lfs -text
7
+ *.gz filter=lfs diff=lfs merge=lfs -text
8
+ *.h5 filter=lfs diff=lfs merge=lfs -text
9
+ *.joblib filter=lfs diff=lfs merge=lfs -text
10
+ *.lfs.* filter=lfs diff=lfs merge=lfs -text
11
+ *.mlmodel filter=lfs diff=lfs merge=lfs -text
12
+ *.model filter=lfs diff=lfs merge=lfs -text
13
+ *.msgpack filter=lfs diff=lfs merge=lfs -text
14
+ *.npy filter=lfs diff=lfs merge=lfs -text
15
+ *.npz filter=lfs diff=lfs merge=lfs -text
16
+ *.onnx filter=lfs diff=lfs merge=lfs -text
17
+ *.ot filter=lfs diff=lfs merge=lfs -text
18
+ *.parquet filter=lfs diff=lfs merge=lfs -text
19
+ *.pb filter=lfs diff=lfs merge=lfs -text
20
+ *.pickle filter=lfs diff=lfs merge=lfs -text
21
+ *.pkl filter=lfs diff=lfs merge=lfs -text
22
+ *.pt filter=lfs diff=lfs merge=lfs -text
23
+ *.pth filter=lfs diff=lfs merge=lfs -text
24
+ *.rar filter=lfs diff=lfs merge=lfs -text
25
+ *.safetensors filter=lfs diff=lfs merge=lfs -text
26
+ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
27
+ *.tar.* filter=lfs diff=lfs merge=lfs -text
28
+ *.tar filter=lfs diff=lfs merge=lfs -text
29
+ *.tflite filter=lfs diff=lfs merge=lfs -text
30
+ *.tgz filter=lfs diff=lfs merge=lfs -text
31
+ *.wasm filter=lfs diff=lfs merge=lfs -text
32
+ *.xz filter=lfs diff=lfs merge=lfs -text
33
+ *.zip filter=lfs diff=lfs merge=lfs -text
34
+ *.zst filter=lfs diff=lfs merge=lfs -text
35
+ *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ tokenizer.json filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,110 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ language:
4
+ - en
5
+ - es
6
+ - pt
7
+ - de
8
+ - fr
9
+ - zh
10
+ - ja
11
+ - ru
12
+ - ko
13
+ pipeline_tag: audio-to-audio
14
+ tags:
15
+ - speech
16
+ - full-duplex
17
+ - speech-to-speech
18
+ - haru
19
+ ---
20
+
21
+ # Haru-4B
22
+
23
+ [Code](https://github.com/higurashi-ai/Haru) ·
24
+ [Technical blog](https://blog.higurashi.ai/melte/haru4b/) ·
25
+ [Technical blog (Japanese)](https://blog.higurashi.ai/melte/haru4b-ja/)
26
+
27
+ Haru-4B is a low-latency and multilingual full-duplex continuous spoken dialogue foundation
28
+ model: it listens and speaks at the same time, the way people do on a phone call. It is a
29
+ pretrained base model, released for research.
30
+
31
+ - **Native full-duplex at 7.5 Hz.** Every 133 ms the model takes one frame of the user's audio
32
+ and produces one frame of its own audio and text. User audio, model audio and model text are
33
+ summed into a single position of the language model, so listening and speaking never wait
34
+ for each other.
35
+ - **Continuous audio, one sampling step.** Audio is a 64-dimensional latent per frame. A small
36
+ MLP head with MeanFlow sampling produces it in one step from the language model's hidden
37
+ state; there is no separate audio transformer and there are no discrete audio tokens.
38
+ - **Character and voice control.** A text card sets the character, a short reference clip sets
39
+ the voice.
40
+ - **Multilingual.** English, Spanish, Portuguese, German, French, Chinese, Japanese, Russian
41
+ and Korean.
42
+
43
+ <p align="center">
44
+ <img src="assets/architecture.svg" width="720" alt="Haru-4B architecture: in every 133 ms frame, the user's speech latent, the model's previous text token and its previous speech latent are summed into one position of the language model, whose output predicts the next text token and the next speech latent.">
45
+ </p>
46
+
47
+ ## Usage
48
+
49
+ The code is at <https://github.com/higurashi-ai/Haru>: a PyTorch implementation with a
50
+ web demo, an MLX implementation with a native macOS app, and a plugin for serving with
51
+ vLLM-Omni.
52
+
53
+ With an NVIDIA GPU (about 12 GB of memory):
54
+
55
+ ```bash
56
+ git clone https://github.com/higurashi-ai/Haru && cd Haru
57
+ pip install -e ./haru
58
+ python -m haru.server --hf-repo higurashi-ai/Haru-4B
59
+ ```
60
+
61
+ Open <http://localhost:8998>, pick a character, give the model a voice, and talk. For low
62
+ latency, we recommend wired headphones and a wired microphone. Bluetooth devices add 100 ms or
63
+ more.
64
+
65
+ **Bring your own voice.** Haru ships without voice clips. Upload or record a clip of the voice
66
+ the model should speak with: one speaker, 3 seconds to 3 minutes, clean speech. Only use voices
67
+ you have the right to use.
68
+
69
+ ```python
70
+ from haru import DuplexSession, HaruModel
71
+ from haru.audio import load_audio, prepare_reference
72
+
73
+ model = HaruModel.from_pretrained("higurashi-ai/Haru-4B")
74
+ voice = prepare_reference(load_audio("voice.wav"))
75
+ session = DuplexSession(model, system_prompt=card, reference=voice)
76
+
77
+ frame = session.start() # the model's first 133 ms: frame.audio, frame.text
78
+ while not frame.done:
79
+ frame = session.step(user_audio) # 3200 samples at 24 kHz in, 3200 out
80
+ ```
81
+
82
+ The system prompt is a character card with a fixed layout; the repository's README shows how
83
+ to build one.
84
+
85
+ ## Files
86
+
87
+ | File | Contents |
88
+ | --- | --- |
89
+ | `model.safetensors` | All weights (bfloat16): `lm.*` language model, `model_audio_proj.*` / `user_audio_proj.*` audio input projections, `head.*` MeanFlow head, `codec.*` audio codec |
90
+ | `config.json` | Model configuration |
91
+ | `tokenizer.json` | Text tokenizer |
92
+
93
+ ## Architecture
94
+
95
+ | Part | Details |
96
+ | --- | --- |
97
+ | Language model | Qwen3.5-4B text backbone, 32 layers, hidden size 2560 |
98
+ | Audio codec | VibeVoice acoustic tokenizer, causal, 24 kHz, 3200 samples per latent |
99
+ | Audio head | 4-layer MLP, MeanFlow, 1 sampling step, guidance scale 1 to 2 (default 1.5) |
100
+ | Frame rate | 7.5 Hz |
101
+
102
+ ## Limitations
103
+
104
+ Haru-4B is a pretrained base model, released for research. It holds a natural conversation,
105
+ but it is not a reliable source of facts and does not follow instructions the way an assistant
106
+ does. It can imitate the voice of a reference clip; only use voices you have the right to use.
107
+
108
+ ## License
109
+
110
+ The weights are released under the Apache License 2.0.
assets/architecture.svg ADDED
config.json ADDED
@@ -0,0 +1,131 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model_type": "haru",
3
+ "architectures": [
4
+ "HaruForConditionalGeneration"
5
+ ],
6
+ "dtype": "bfloat16",
7
+ "sample_rate": 24000,
8
+ "frame_samples": 3200,
9
+ "frame_rate": 7.5,
10
+ "latent_dim": 64,
11
+ "pad_token_id": 248076,
12
+ "eop_token_id": 248071,
13
+ "lm": {
14
+ "model_type": "qwen3_5_text",
15
+ "vocab_size": 248320,
16
+ "hidden_size": 2560,
17
+ "intermediate_size": 9216,
18
+ "num_hidden_layers": 32,
19
+ "num_attention_heads": 16,
20
+ "num_key_value_heads": 4,
21
+ "head_dim": 256,
22
+ "hidden_act": "silu",
23
+ "max_position_embeddings": 262144,
24
+ "rms_norm_eps": 1e-06,
25
+ "attention_bias": false,
26
+ "attention_dropout": 0.0,
27
+ "attn_output_gate": true,
28
+ "full_attention_interval": 4,
29
+ "layer_types": [
30
+ "linear_attention",
31
+ "linear_attention",
32
+ "linear_attention",
33
+ "full_attention",
34
+ "linear_attention",
35
+ "linear_attention",
36
+ "linear_attention",
37
+ "full_attention",
38
+ "linear_attention",
39
+ "linear_attention",
40
+ "linear_attention",
41
+ "full_attention",
42
+ "linear_attention",
43
+ "linear_attention",
44
+ "linear_attention",
45
+ "full_attention",
46
+ "linear_attention",
47
+ "linear_attention",
48
+ "linear_attention",
49
+ "full_attention",
50
+ "linear_attention",
51
+ "linear_attention",
52
+ "linear_attention",
53
+ "full_attention",
54
+ "linear_attention",
55
+ "linear_attention",
56
+ "linear_attention",
57
+ "full_attention",
58
+ "linear_attention",
59
+ "linear_attention",
60
+ "linear_attention",
61
+ "full_attention"
62
+ ],
63
+ "linear_conv_kernel_dim": 4,
64
+ "linear_key_head_dim": 128,
65
+ "linear_value_head_dim": 128,
66
+ "linear_num_key_heads": 16,
67
+ "linear_num_value_heads": 32,
68
+ "mamba_ssm_dtype": "float32",
69
+ "partial_rotary_factor": 0.25,
70
+ "rope_parameters": {
71
+ "mrope_interleaved": true,
72
+ "mrope_section": [
73
+ 11,
74
+ 11,
75
+ 10
76
+ ],
77
+ "partial_rotary_factor": 0.25,
78
+ "rope_theta": 10000000,
79
+ "rope_type": "default"
80
+ },
81
+ "tie_word_embeddings": true,
82
+ "initializer_range": 0.02,
83
+ "eos_token_id": 248044
84
+ },
85
+ "codec": {
86
+ "latent_dim": 64,
87
+ "base_channels": 32,
88
+ "ratios": [
89
+ 8,
90
+ 5,
91
+ 5,
92
+ 4,
93
+ 2,
94
+ 2
95
+ ],
96
+ "encoder_depths": [
97
+ 3,
98
+ 3,
99
+ 3,
100
+ 3,
101
+ 3,
102
+ 3,
103
+ 8
104
+ ],
105
+ "decoder_depths": [
106
+ 8,
107
+ 3,
108
+ 3,
109
+ 3,
110
+ 3,
111
+ 3,
112
+ 3
113
+ ],
114
+ "norm_eps": 1e-05,
115
+ "latent_std": 0.5
116
+ },
117
+ "head": {
118
+ "hidden_size": 2560,
119
+ "latent_dim": 64,
120
+ "num_layers": 4,
121
+ "ffn_ratio": 3.0,
122
+ "norm_eps": 1e-05,
123
+ "interval_embed": 7.0,
124
+ "num_steps": 1,
125
+ "guidance_scale": 1.5,
126
+ "guidance_range": [
127
+ 1.0,
128
+ 2.0
129
+ ]
130
+ }
131
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ad2c3534ef6bd88525e22e5da3ac4fb739368406bf3b1c852cbafbade647c0fa
3
+ size 10540113510
tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:06b9509352d2af50381ab2247e083b80d32d5c0aba91c272ca9ff729b6a0e523
3
+ size 19989325
tokenizer_config.json ADDED
@@ -0,0 +1,14 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": false,
3
+ "backend": "tokenizers",
4
+ "bos_token": null,
5
+ "clean_up_tokenization_spaces": false,
6
+ "eos_token": "<|im_end|>",
7
+ "errors": "replace",
8
+ "model_max_length": 262144,
9
+ "pad_token": "<|endoftext|>",
10
+ "pretokenize_regex": "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?[\\p{L}\\p{M}]+|\\p{N}| ?[^\\s\\p{L}\\p{M}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+",
11
+ "split_special_tokens": false,
12
+ "tokenizer_class": "Qwen2Tokenizer",
13
+ "unk_token": null
14
+ }