README.md

---
language:
- en
license: apache-2.0
tags:
- text-to-speech
- tts
- audio-language-model
- voice-cloning
- zero-shot-tts
- neurovoice
- qwen2.5
- mimi-codec
- pytorch
- deep-learning
datasets:
- parler-tts/libritts_r_filtered
pipeline_tag: text-to-speech
inference: false
---

# 🎙️ NeuroVoice-0.5B (v0.1 Alpha)

<p align="center">
  <img src="https://img.shields.io/badge/Model-NeuroVoice--0.5B-7928CA?style=for-the-badge&logo=openai" alt="Model">
  <img src="https://img.shields.io/badge/Architecture-Two--Stage%20Audio%20LM-blue?style=for-the-badge" alt="Architecture">
  <img src="https://img.shields.io/badge/Backbone-Qwen2.5--0.5B-green?style=for-the-badge" alt="Backbone">
  <img src="https://img.shields.io/badge/Audio%20Codec-Kyutai%20Mimi%2024kHz-orange?style=for-the-badge" alt="Codec">
  <img src="https://img.shields.io/badge/Hardware-4x%20NVIDIA%20H100-red?style=for-the-badge" alt="Hardware">
  <img src="https://img.shields.io/badge/License-Apache%202.0-yellow?style=for-the-badge" alt="License">
</p>

**NeuroVoice-0.5B (v0.1 Alpha)** is an original, autoregressive **Neural Audio Language Model** built from the ground up for expressive text-to-speech and zero-shot voice cloning. It couples the deep linguistic reasoning of **Qwen2.5-0.5B** with the high-compression **Kyutai Mimi** 24 kHz neural audio codec (12.5 Hz frame rate, 8 codebooks).

The model was developed, optimized, and trained on **4x NVIDIA H100 GPUs** (DistributedDataParallel) using PyTorch FlashAttention-2 / SDPA, delivering over **99,800 tokens/sec** training throughput.

---

## 🏗️ NeuroVoice Architecture

NeuroVoice operates through a **Two-Stage Autoregressive Backbone + Depth Decoder** hierarchy:

```
[Text Prompt + Reference Audio]
             │
             ▼
┌─────────────────────────────────────────────────────────────┐
│ Stage 1: Main Audio Transformer (Qwen2.5-0.5B Backbone)    │
│  - 24 Transformer Layers (d_model=896, 14 Heads, 2 KV Heads)│
│  - SwiGLU MLP (d_ff=4864), RoPE, RMSNorm                    │
│  - Autoregressively predicts Codebook 0 (Semantic & Pitch)  │
└─────────────────────────────────────────────────────────────┘
             │
             ▼  (Audio Hidden States + CB0 Tokens)
┌─────────────────────────────────────────────────────────────┐
│ Stage 2: 4-Layer Depth Decoder (Acoustic Hierarchy)         │
│  - Hierarchically generates Codebooks 1..7                  │
│  - Reconstructs high-frequency acoustics & vocal timbre     │
└─────────────────────────────────────────────────────────────┘
             │
             ▼  (Full 8-Codebook Matrix @ 12.5 Hz)
┌─────────────────────────────────────────────────────────────┐
│ Kyutai Mimi Neural Audio Codec Decoder                      │
│  - Synthesizes 24 kHz Hi-Fi Audio Waveform                  │
└─────────────────────────────────────────────────────────────┘
```

* **Codec Details:** Kyutai Mimi compresses 24,000 samples/sec audio into **12.5 frames per second** with **8 codebooks** (vocabulary size 2048 per codebook), achieving an extreme 1920x temporal compression.
* **Stage 1 (Main Backbone):** Autoregressively predicts the speech backbone (Codebook 0).
* **Stage 2 (Depth Decoder):** Takes backbone hidden representations and CB0 to hierarchically generate the remaining 7 acoustic codebooks (CB1 to CB7) with soft acoustic sampling.

---

## 🍳 Training Recipe & Hyperparameters

Trained on MareNostrum 5 using PyTorch DistributedDataParallel (DDP):

| Hyperparameter | Value |
| :--- | :--- |
| **Compute Hardware** | 1 Node / 4x NVIDIA H100 (80GB HBM3) |
| **Dataset** | `parler-tts/libritts_r_filtered` (Clean subset, 21,000 samples, ~35.2 hours) |
| **Global Batch Size** | 48 (12 per GPU x 4 GPUs) |
| **Optimizer** | AdamW ($\beta_1=0.9, \beta_2=0.95, \text{weight\_decay}=0.1$) |
| **Learning Rate** | $2.0 \times 10^{-4}$ (Cosine Annealing schedule) |
| **Warmup Steps** | 200 steps (Linear warmup from $1.0 \times 10^{-6}$) |
| **Precision** | `bfloat16` Mixed Precision (`torch.autocast`) |
| **Attention Kernel** | PyTorch `F.scaled_dot_product_attention` (FlashAttention-2) |
| **Total Epochs / Steps** | 15 Epochs / 6,555 Steps |
| **Training Duration** | **23 minutes 52 seconds** |
| **Peak Throughput** | **~99,800 tokens / second** |

### 📉 Loss Trajectory Across Training

```
Step 0000: Total Loss: 25.1857 | CB0 Loss: 17.3926 | Depth Loss: 7.7931
Step 1000: Total Loss:  3.1959 | CB0 Loss:  0.6863 | Depth Loss: 2.5096
Step 3000: Total Loss:  2.2011 | CB0 Loss:  0.1136 | Depth Loss: 2.0875
Step 5000: Total Loss:  1.4281 | CB0 Loss:  0.0029 | Depth Loss: 1.4252
Step 6555: Total Loss:  1.0407 | CB0 Loss:  0.0034 | Depth Loss: 1.0372 (Best Epoch Avg: 1.1138)
```

---

## 🚀 Quickstart & Inference

### 1. Installation

```bash
pip install torch transformers soundfile torchaudio
```

### 2. Loading with Hugging Face `AutoModel` (Recommended)

Thanks to `AutoConfig` and `AutoModel` integration (`trust_remote_code=True`), you can load NeuroVoice directly in Python without cloning the repository:

```python
import torch
from transformers import AutoConfig, AutoModel

repo_id = "TurkishCodeMan/NeuroVoice-0.5B"

# 1. Konfigürasyonu ve Modeli Yükle (Safetensors / bfloat16)
config = AutoConfig.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModel.from_pretrained(repo_id, trust_remote_code=True)
model = model.to("cuda" if torch.cuda.is_available() else "cpu")
model.eval()

print(f"✅ {config.model_name} başarıyla yüklendi!")
print(f"Parametre Sayısı: {sum(p.numel() for p in model.parameters()) / 1e6:.2f}M")
```

---

### 3. Standalone CLI Inference (Cloned Repository)

You can also clone the repository and use the built-in `inference.py` script:

```bash
git clone https://huggingface.co/TurkishCodeMan/NeuroVoice-0.5B
cd NeuroVoice-0.5B
```

#### A. Standard Text-to-Speech
```bash
python inference.py \
    --use_qwen_backbone \
    --checkpoint model.safetensors \
    --text "Artificial intelligence is creating the future of speech synthesis." \
    --output output_tts.wav \
    --temperature 0.6 \
    --depth_temperature 0.6 \
    --max_tokens 80
```

#### B. Zero-Shot Voice Cloning
Use a 3 to 4-second clean audio sample (`.wav`) as reference:

```bash
python inference.py \
    --use_qwen_backbone \
    --checkpoint model.safetensors \
    --ref_audio reference_voice.wav \
    --ref_max_sec 3.5 \
    --text "Hello! This speech is synthesized using the NeuroVoice neural audio model." \
    --output output_cloned.wav \
    --temperature 0.5 \
    --depth_temperature 0.5 \
    --max_tokens 80
```

---

## 🔬 Dataset Analysis & Limitations

The v0.1 model was trained on **35.17 hours** of speech from the LibriTTS filtered clean subset:
* **Total Samples:** 21,000 audio-text pairs
* **Average Sentence Duration:** 6.03 seconds (18 words, ~4.2 codec frames per word)
* **Reference Audio Range in Training:** 2.0s to 4.0s (Average 3.71s)

### ⚠️ Known Limitations in v0.1 Alpha:
1. **Audiobook Monotony:** LibriTTS is narrated audiobook data; speech has a formal, studio-reader cadence rather than conversational prosody.
2. **Optimal Reference Audio:** The model was trained with 2-4 second reference clips. Use `--ref_max_sec 3.5` for optimal voice cloning quality.
3. **Acoustic Timbre:** Depth Decoder loss converged to ~1.0, which produces intelligible speech with a slightly synthetic timbre. Scaling up to larger multi-speaker datasets in v0.2 will address this.

---

## 🗺️ Roadmap (v0.2 & Beyond)

- [ ] **Scale Training Data:** Expand from 35 hours to 500+ hours with diverse conversational audio.
- [ ] **Enhanced Depth Decoder:** Increase Depth Decoder capacity from 4 to 8 layers for warmer human timbre.
- [ ] **Multilingual NeuroVoice:** Extend tokenization to Turkish and multilingual speech.

---

## 📜 Citation & License

This project is licensed under the **Apache 2.0 License**.

```bibtex
@misc{neurovoice-0.5b-2026,
  author = {TurkishCodeMan},
  title = {NeuroVoice-0.5B: An Autoregressive Neural Audio Language Model},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/TurkishCodeMan/NeuroVoice-0.5B}}
}
```