🧠 AphaVoice: Text-to-Speech Model for Aphasia Patient Simulation

AphaVoice is a specialized Text-to-Speech (TTS) model designed to simulate non-fluent aphasic speech for medical student training and clinical simulation. AphaVoice fine-tunes the non-autoregressive architecture of Matcha-TTS to accurately map the distinct temporal and phonetic dysfluencies characteristic of post-stroke aphasia.

🚀 Key Features

• Pathological Realism: Outperforms off-the-shelf TTS models in accurately simulating aphasic Words-Per-Minute (WPM) and pathological pause ratios.
• Acoustic Stability: Achieves high Speaker Encoder Cosine Similarity (SECS) scores, ensuring the underlying voice profile remains perfectly consistent despite severe simulated dysfluency.
• Multi-Speaker Latent Shift: Supports dynamic latent shifting to generate both Male and Female baseline speaker profiles.
• Clinical Dysfluency Engine: Includes a custom linguistic pipeline to inject mathematically weighted phonetic false starts, repetitions, and conversational fillers prior to acoustic generation.

💻 Quick Start (Python)

You can download and run the model locally using the huggingface_hub library. Before you could use the model, as this is built on top of Matcha-TTS, you need to install all their dependencies to make it work. Please refer to this github: https://github.com/varrelkusuma/AphaVoice

Prerequisites:

pip install huggingface_hub soundfile spacy pyyaml torch safetensors noisereduce
python -m spacy download en_core_web_sm
sudo apt-get install espeak-ng  # Required for Linux/Colab phonemization

Inference Code:

import os
import sys
import soundfile as sf
import IPython.display as ipd
from huggingface_hub import snapshot_download

# 1. Download the AphaVoice repository
repo_path = snapshot_download(repo_id="varrelkusuma/AphaVoice")
sys.path.insert(0, repo_path)
from inference import AphaVoicePipeline

# 2. Initialize the model
pipeline = AphaVoicePipeline(
    config_path=os.path.join(repo_path, "config.yaml"),
    model_path=os.path.join(repo_path, "model.ckpt"),
    vocoder_path=os.path.join(repo_path, "vocoder")
)

# 3. Generate Speech
text = "The patient needs a glass of water."
sample_rate, audio_data, final_script = pipeline.synthesise(
    text=text,
    speaker_id=0,           # 0 for Female Model, 1 for Male Model
    apply_aphasia=True,     # Applies the clinical dysfluency script, change to "False" for normal speech generation
    severity=0.4,           # Probability of dysfluency, value from 0-1
    length_scale=1.0        # Pacing adjustment
)

# 4. Save output
print(f"Synthesized script: {final_script}")
display(ipd.Audio(audio_data, rate=22050))

📝 Citation

If you use AphaVoice in your research, please cite our paper:

@misc{kusuma2026aphavoicetexttospeechmodelaphasia,
      title={AphaVoice: Text-to-Speech Model for Aphasia Patient Simulation}, 
      author={Al Varrel Putra Kusuma and Fernando Bello and Joshua Brown},
      year={2026},
      eprint={2610.14479},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2610.14479}, 
}