OmniVoice Word-Control

OmniVoice (0.6B masked-diffusion LM TTS on Qwen3-0.6B) fine-tuned for five-dimensional word-level acoustic control, following the task of WordVoice (arXiv:2607.06461) and its dataset WordVoice-5A.

Control is injected as inline special tokens placed before any word (any subset works; untagged words are planned freely by the model):

<|dur_9|><|bnd_0|><|eng_15|><|pit_17|><|ton_rise|>word

tokens	attribute	discretization
`<\|dur_0..63\|>`	word duration	40 ms frames (40 ms – 2.56 s)
`<\|bnd_0..4\|>`	pause after word	b0 (none) … b4 (long, >0.4 s)
`<\|eng_0..19\|>`	energy	20 uniform bins over [0, 1]
`<\|pit_0..19\|>`	pitch (core F0)	20 uniform bins over [−1, 1]
`<\|ton_flat\|rise\|rrise\|fall\|ffall\|peak\|valley\|>`	pitch contour	7 morphologies

Usage

from omnivoice import OmniVoice  # code: github.com/k2-fsa/OmniVoice

model = OmniVoice.from_pretrained("multimodalart/omnivoice-word-control", device_map="cuda")
audio = model.generate(
    text="I will <|pit_3|><|ton_ffall|>never agree to <|bnd_4|>this!",
    language="en",
    ref_audio="ref.wav",
    ref_text="Transcript of the reference clip.",
)

The tokenizer in this repo already contains the 116 control tokens. For strict overall timing, also pass duration= (the diffusion canvas is fixed before generation, so word duration tokens redistribute time within it). Try it in the demo Space.

Training

Init from k2-fsa/OmniVoice; all parameters trained; masked-diffusion objective unchanged (control tokens are conditioning only — no loss on text).
Data: WordVoice-5A English, all 4 train shards (~2,138 h, 1.18 M utterances); on-the-fly tag construction with dropout (15% utterances untagged / 20% words / attributes kept @ 0.7) to preserve plain TTS and enable partial control.
40k steps × 8,192 effective tokens (~1 epoch), lr 5e-5 cosine, bf16, single A100-40GB, ≈6 h.
Final eval loss 4.109 (best of run). Directional probes (hi/lo tag ratio, seed-matched): pitch 3.61, energy 2.52, duration 2.70.

Fine-tuning recipe: the word-control patch set on the OmniVoice trainer (omnivoice/data/word_control.py — tag vocabulary, binning, alignment, dropout).

License & lineage

Weights are CC-BY-NC-4.0 (derivative of the CC-BY-NC OmniVoice release). Data: WordVoice-5A (CC-BY-4.0). Task formulation: WordVoice, arXiv:2607.06461.