Upload README.md with huggingface_hub

9bde6c7 verified 12 days ago

8.98 kB

	---
	language:
	- en
	license: apache-2.0
	tags:
	- text-to-speech
	- tts
	- audio-language-model
	- voice-cloning
	- zero-shot-tts
	- neurovoice
	- qwen2.5
	- mimi-codec
	- pytorch
	- deep-learning
	datasets:
	- parler-tts/libritts_r_filtered
	pipeline_tag: text-to-speech
	inference: false
	---

	# 🎙️ NeuroVoice-0.5B (v0.1 Alpha)

	<p align="center">
	<img src="https://img.shields.io/badge/Model-NeuroVoice--0.5B-7928CA?style=for-the-badge&logo=openai" alt="Model">
	<img src="https://img.shields.io/badge/Architecture-Two--Stage%20Audio%20LM-blue?style=for-the-badge" alt="Architecture">
	<img src="https://img.shields.io/badge/Backbone-Qwen2.5--0.5B-green?style=for-the-badge" alt="Backbone">
	<img src="https://img.shields.io/badge/Audio%20Codec-Kyutai%20Mimi%2024kHz-orange?style=for-the-badge" alt="Codec">
	<img src="https://img.shields.io/badge/Hardware-4x%20NVIDIA%20H100-red?style=for-the-badge" alt="Hardware">
	<img src="https://img.shields.io/badge/License-Apache%202.0-yellow?style=for-the-badge" alt="License">
	</p>

	NeuroVoice-0.5B (v0.1 Alpha) is an original, autoregressive Neural Audio Language Model built from the ground up for expressive text-to-speech and zero-shot voice cloning. It couples the deep linguistic reasoning of Qwen2.5-0.5B with the high-compression Kyutai Mimi 24 kHz neural audio codec (12.5 Hz frame rate, 8 codebooks).

	The model was developed, optimized, and trained on 4x NVIDIA H100 GPUs (DistributedDataParallel) using PyTorch FlashAttention-2 / SDPA, delivering over 99,800 tokens/sec training throughput.

	---

	## 🏗️ NeuroVoice Architecture

	NeuroVoice operates through a Two-Stage Autoregressive Backbone + Depth Decoder hierarchy:

	```
	[Text Prompt + Reference Audio]
	│
	▼
	┌─────────────────────────────────────────────────────────────┐
	│ Stage 1: Main Audio Transformer (Qwen2.5-0.5B Backbone) │
	│ - 24 Transformer Layers (d_model=896, 14 Heads, 2 KV Heads)│
	│ - SwiGLU MLP (d_ff=4864), RoPE, RMSNorm │
	│ - Autoregressively predicts Codebook 0 (Semantic & Pitch) │
	└─────────────────────────────────────────────────────────────┘
	│
	▼ (Audio Hidden States + CB0 Tokens)
	┌─────────────────────────────────────────────────────────────┐
	│ Stage 2: 4-Layer Depth Decoder (Acoustic Hierarchy) │
	│ - Hierarchically generates Codebooks 1..7 │
	│ - Reconstructs high-frequency acoustics & vocal timbre │
	└─────────────────────────────────────────────────────────────┘
	│
	▼ (Full 8-Codebook Matrix @ 12.5 Hz)
	┌─────────────────────────────────────────────────────────────┐
	│ Kyutai Mimi Neural Audio Codec Decoder │
	│ - Synthesizes 24 kHz Hi-Fi Audio Waveform │
	└─────────────────────────────────────────────────────────────┘
	```

	* Codec Details: Kyutai Mimi compresses 24,000 samples/sec audio into 12.5 frames per second with 8 codebooks (vocabulary size 2048 per codebook), achieving an extreme 1920x temporal compression.
	* Stage 1 (Main Backbone): Autoregressively predicts the speech backbone (Codebook 0).
	* Stage 2 (Depth Decoder): Takes backbone hidden representations and CB0 to hierarchically generate the remaining 7 acoustic codebooks (CB1 to CB7) with soft acoustic sampling.

	---

	## 🍳 Training Recipe & Hyperparameters

	Trained on MareNostrum 5 using PyTorch DistributedDataParallel (DDP):

	\| Hyperparameter \| Value \|
	\| :--- \| :--- \|
	\| Compute Hardware \| 1 Node / 4x NVIDIA H100 (80GB HBM3) \|
	\| Dataset \| `parler-tts/libritts_r_filtered` (Clean subset, 21,000 samples, ~35.2 hours) \|
	\| Global Batch Size \| 48 (12 per GPU x 4 GPUs) \|
	\| Optimizer \| AdamW ($\beta_1=0.9, \beta_2=0.95, \text{weight\_decay}=0.1$) \|
	\| Learning Rate \| $2.0 \times 10^{-4}$ (Cosine Annealing schedule) \|
	\| Warmup Steps \| 200 steps (Linear warmup from $1.0 \times 10^{-6}$) \|
	\| Precision \| `bfloat16` Mixed Precision (`torch.autocast`) \|
	\| Attention Kernel \| PyTorch `F.scaled_dot_product_attention` (FlashAttention-2) \|
	\| Total Epochs / Steps \| 15 Epochs / 6,555 Steps \|
	\| Training Duration \| 23 minutes 52 seconds \|
	\| Peak Throughput \| ~99,800 tokens / second \|

	### 📉 Loss Trajectory Across Training

	```
	Step 0000: Total Loss: 25.1857 \| CB0 Loss: 17.3926 \| Depth Loss: 7.7931
	Step 1000: Total Loss: 3.1959 \| CB0 Loss: 0.6863 \| Depth Loss: 2.5096
	Step 3000: Total Loss: 2.2011 \| CB0 Loss: 0.1136 \| Depth Loss: 2.0875
	Step 5000: Total Loss: 1.4281 \| CB0 Loss: 0.0029 \| Depth Loss: 1.4252
	Step 6555: Total Loss: 1.0407 \| CB0 Loss: 0.0034 \| Depth Loss: 1.0372 (Best Epoch Avg: 1.1138)
	```

	---

	## 🚀 Quickstart & Inference

	### 1. Installation

	```bash
	pip install torch transformers soundfile torchaudio
	```

	### 2. Loading with Hugging Face `AutoModel` (Recommended)

	Thanks to `AutoConfig` and `AutoModel` integration (`trust_remote_code=True`), you can load NeuroVoice directly in Python without cloning the repository:

	```python
	import torch
	from transformers import AutoConfig, AutoModel

	repo_id = "TurkishCodeMan/NeuroVoice-0.5B"

	# 1. Konfigürasyonu ve Modeli Yükle (Safetensors / bfloat16)
	config = AutoConfig.from_pretrained(repo_id, trust_remote_code=True)
	model = AutoModel.from_pretrained(repo_id, trust_remote_code=True)
	model = model.to("cuda" if torch.cuda.is_available() else "cpu")
	model.eval()

	print(f"✅ {config.model_name} başarıyla yüklendi!")
	print(f"Parametre Sayısı: {sum(p.numel() for p in model.parameters()) / 1e6:.2f}M")
	```

	---

	### 3. Standalone CLI Inference (Cloned Repository)

	You can also clone the repository and use the built-in `inference.py` script:

	```bash
	git clone https://huggingface.co/TurkishCodeMan/NeuroVoice-0.5B
	cd NeuroVoice-0.5B
	```

	#### A. Standard Text-to-Speech
	```bash
	python inference.py \
	--use_qwen_backbone \
	--checkpoint model.safetensors \
	--text "Artificial intelligence is creating the future of speech synthesis." \
	--output output_tts.wav \
	--temperature 0.6 \
	--depth_temperature 0.6 \
	--max_tokens 80
	```

	#### B. Zero-Shot Voice Cloning
	Use a 3 to 4-second clean audio sample (`.wav`) as reference:

	```bash
	python inference.py \
	--use_qwen_backbone \
	--checkpoint model.safetensors \
	--ref_audio reference_voice.wav \
	--ref_max_sec 3.5 \
	--text "Hello! This speech is synthesized using the NeuroVoice neural audio model." \
	--output output_cloned.wav \
	--temperature 0.5 \
	--depth_temperature 0.5 \
	--max_tokens 80
	```

	---

	## 🔬 Dataset Analysis & Limitations

	The v0.1 model was trained on 35.17 hours of speech from the LibriTTS filtered clean subset:
	* Total Samples: 21,000 audio-text pairs
	* Average Sentence Duration: 6.03 seconds (18 words, ~4.2 codec frames per word)
	* Reference Audio Range in Training: 2.0s to 4.0s (Average 3.71s)

	### ⚠️ Known Limitations in v0.1 Alpha:
	1. Audiobook Monotony: LibriTTS is narrated audiobook data; speech has a formal, studio-reader cadence rather than conversational prosody.
	2. Optimal Reference Audio: The model was trained with 2-4 second reference clips. Use `--ref_max_sec 3.5` for optimal voice cloning quality.
	3. Acoustic Timbre: Depth Decoder loss converged to ~1.0, which produces intelligible speech with a slightly synthetic timbre. Scaling up to larger multi-speaker datasets in v0.2 will address this.

	---

	## 🗺️ Roadmap (v0.2 & Beyond)

	- [ ] Scale Training Data: Expand from 35 hours to 500+ hours with diverse conversational audio.
	- [ ] Enhanced Depth Decoder: Increase Depth Decoder capacity from 4 to 8 layers for warmer human timbre.
	- [ ] Multilingual NeuroVoice: Extend tokenization to Turkish and multilingual speech.

	---

	## 📜 Citation & License

	This project is licensed under the Apache 2.0 License.

	```bibtex
	@misc{neurovoice-0.5b-2026,
	author = {TurkishCodeMan},
	title = {NeuroVoice-0.5B: An Autoregressive Neural Audio Language Model},
	year = {2026},
	publisher = {Hugging Face},
	howpublished = {\url{https://huggingface.co/TurkishCodeMan/NeuroVoice-0.5B}}
	}
	```