VoxDance-Lab/VoxMel-0.3B

VoxMel

VoxMel is a tokenizer-free continuous autoregressive text-to-speech (TTS) framework that directly models mel-spectrograms through a fixed, exactly invertible two-axis Haar transform. A language-model backbone with a flow-matching head generates Coarse-Mel autoregressively; two non-autoregressive residual tiers recover temporal and frequency details without feeding them back into the autoregressive history. A separate Vocos vocoder converts the reconstructed mel-spectrogram into speech.

Highlights

• Direct mel modeling — No learned acoustic tokenizer or compressed acoustic latent space; the text frontend uses the Qwen2.5 tokenizer.
• Exactly invertible decomposition — Frequency and temporal Haar splits preserve all mel coefficients. Exact invertibility applies to the transform, rather than guaranteeing lossless generated audio or vocoder reconstruction.
• Coarse-only autoregression — Generate Coarse-Mel first, then the temporal residual and frequency residual in dependency order. Residual details stay outside the autoregressive history.
• English and Chinese zero-shot cloning — Synthesize speech from reference audio and its transcript, with 24 kHz output in the paper's configuration.
• Post-training — Frozen-head GRPO improves all four Seed-TTS metrics.

Quick Start

Installation

Use Python 3.10–3.12. Install matching PyTorch and torchaudio builds for your hardware (the original environment uses torch/torchaudio 2.4.1 with CUDA 12.1), then install from the repository root:

pip install -e .

requirements.txt is shared by the package installer. Vocos is included. For CPU or Apple Silicon, use the appropriate PyTorch build. device="auto" selects CUDA, then MPS, then CPU; an unavailable explicit device raises an error. MPS operator support and performance still require validation with your checkpoint.

Prepare Model Files

This checkout includes model code and a Qwen tokenizer, but no trained VoxMel weights, experiment config, or Vocos weights. Supply a compatible experiment:

/path/to/experiment/
├── config.yaml
└── model_M_final.pt
/path/to/vocos-mel-24khz/
├── config.yaml
└── pytorch_model.bin

Python API

from voxcore import VoxMel
import soundfile as sf

model = VoxMel.from_pretrained(
    "/path/to/experiment",
    tokenizer_path="examples/voxmel/qwen_tokenizer",
    vocos_path="/path/to/vocos-mel-24khz",
    device="auto",
)

# Audio continuation cloning requires the exact reference transcript.
wav = model.generate(
    "这是声音克隆的演示。",
    prompt_wav_path="reference.wav",
    prompt_text="参考音频中实际说出的内容。",
    cfg_text_strength=3.5,
    cfg_spk_strength=3.0,
    seed=42,
)
sf.write("clone.wav", wav, model.sample_rate)

The API returns a mono float32 NumPy waveform and runs in inference mode. min_seconds and max_seconds control coarse-frame generation bounds; model stop predictions affect actual duration. cfg_text_strength and cfg_spk_strength independently control text and speaker guidance, while up_num_steps sets residual solver steps. Speaker guidance applies only when reference audio is provided.

License

Released under the Apache-2.0 license, free for commercial use. For production deployments, we recommend thorough testing and safety evaluation tailored to your use case.