README.md

metadata

language:
  - en
license: cc-by-nc-4.0
pipeline_tag: text-to-speech
tags:
  - full-duplex
  - conversation-tts
  - dialogue
  - stereo
  - xml
  - qwen3-tts
base_model: Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice
datasets:
  - facebook/seamless-interaction

DuDE: Full-Duplex Conversation TTS

DuDE is a full-duplex conversation TTS (text-to-speech) model that generates two speakers together from an XML dialogue. It supports turn-taking, backchannels, and overlapping speech, with a separate voice for each speaker. The output is a 24 kHz stereo WAV: speaker A on the left, speaker B on the right.

Author: Cheng-Kuang Chang

Training details

DuDE first self-distills Qwen3-TTS on independent utterances to adapt a shared backbone to two interleaved channels. It is then fine-tuned on approximately 1,922 hours of synchronized Seamless Interaction conversations to learn turn-taking, backchannels, and overlapping speech.

The XML frontend resolves turn dependencies into interleaved text inputs. Reference-speaker embeddings condition the two voices, and each audio channel has its own end-of-speech decision while sharing the same timeline.

Audio samples

The samples use XML transcripts of conversations from Meta's Seamless Interaction dataset, produced using ASR and vocal-event annotation. The audio is generated by DuDE.

Sample 1

Download stereo audio · Input XML · Voices: voice_1 / voice_2

Show complete input XML

<duplex>
<B>So, <e1/></B>
<B start="e1">if you had a chance to <e2/></B>
<B start="e2">send a message back to <e3/> the future, or the past, <e4/></B>
<A start="e3"><sneeze/> <cough/></A>
<B start="e4">or reverse, or whatever, however, what would you do? <e5/> Well, if I had a chance, I probably</B>
<A start="e5">Well, if I had a chance, I'd probably. <e6/></A>
<A start="e6"><e7/> Where can I do both, or I have to do one or the other? <e8/></A>
<B start="e7">would do both. Probably,</B>
<B start="e8">no, you probably will do both regardless. <e9/></B>
<A start="e9">Well, most <e10/> likely, I'd send a message back to the past and be like, "Yo, <e11/> watch out! Don't go around that corner." <cough/> <e12/></A>
<B start="e10">I most likely would</B>
<B start="e11"><cough/></B>
<B start="e12">who? <e13/> <laugh/></B>
<A start="e13">I would send a message to myself. <e14/></A>
<B start="e14">Oh, cool. <e15/></B>
<A start="e15">I mean, who else would I contact in the past? <e16/></A>
<B start="e16">Would you like yourself back then? <e17/></B>
<A start="e17"><e18/> I don't listen. I have no fucking clue. <cough/> <e19/></A>
<B start="e18"><cough/></B>
<B start="e19">Okay. That far. Oh, great. <e20/> great! <e21/></B>
<A start="e20">I don't have a clue.</A>
<B start="e21"><gasp/> <sigh/> <sneeze/> Bye.</B>
</duplex>

Sample 2

Download stereo audio · Input XML · Voices: voice_3 / voice_4

Show complete input XML

<duplex>
<B>If we're to organize a fitness challenge, which <e1/></B>
<B start="e1">I know you love, <e2/></B>
<B start="e2">what would we include <e3/></B>
<B start="e3">for activities? <e4/></B>
<B start="e4">How would we make it engaging? <e5/></B>
<B start="e5">What <e6/> do you think?</B>
<A start="e6">The activities number one has got to be yoga, <e7/> probably.</A>
<B start="e7">Yoga. I'm on <e8/> board. <e9/></B>
<A start="e8">Stretching and</A>
<A start="e9">you know, relaxing and feeling good. <e10/></A>
<A start="e10">I I think weights are good too. I mean, just a little everything. <e11/></A>
<A start="e11">I think one extreme to the other, maybe. <e12/></A>
<B start="e12">Yeah, I mean, I'm definitely on board with yoga. I like mobility exercises. I <e13/></B>
<A start="e13">Yeah, I think if you <e14/> do,</A>
<B start="e14">don't know if I like the word extreme, but <e15/> you know.</B>
<A start="e15">yeah. <e16/></A>
<A start="e16">So the only thing missing would be so you have yoga, weights, and we need some kind of cardio. <e17/></A>
<A start="e17"><e18/> Any suggestions? <e19/></A>
<B start="e18">Yeah.</B>
<B start="e19">Like speed walking, I guess. <e20/></B>
<A start="e20">Maybe a treadmill. <e21/></A>
<B start="e21">Yeah. Yeah, I like <e22/> the treadmill.</B>
<A start="e22">Yeah, I think that would cover everything.</A>
</duplex>

Download and run

Use Python 3.12 and a CUDA-capable NVIDIA GPU. The download includes the model, audio codec, XML frontend, four voice templates, and inference code. Requirements pin PyTorch and torchaudio 2.8.0; use the corresponding CUDA builds for your system.

pip install "huggingface_hub>=0.34,<1"
hf download penguinfish1688/duplexdataengine --local-dir DuDE
cd DuDE
pip install -r requirements.txt
python infer.py \
  --model . --xml examples/sample_1.xml \
  --voice-a voice_1 --voice-b voice_2 \
  --seed 20261060 --max-seconds 118 \
  --output dialogue.wav

Or use the Python API from the downloaded directory:

from pathlib import Path
from dude_tts import DuDE

model = DuDE.from_pretrained(".", device="cuda")
audio = model.generate(
    Path("examples/sample_1.xml").read_text(),
    voice_a="voice_1", voice_b="voice_2",
    seed=20261060, max_seconds=118,
)
audio.save("dialogue.wav")

max_seconds is an output limit, not a target speaking duration. Generation finishes when both speakers emit an end-of-speech token; audio.eos reports whether each speaker finished. Leading silence and pauses are preserved, with silence padding after the shorter channel ends. CPU inference is available with --device cpu, but is slow.

Preset inference on an RTX PRO 6000 Blackwell used 11.6 GB of allocated GPU memory and took about 1.1 seconds per second of stereo output. Weights are stored in FP32; CUDA inference uses BF16 autocast.

Writing a dialogue

Use <A> and <B> for the speakers. Exactly one initial turn has no start attribute. Every later turn references an inline anchor with start="e1", start="e2", and so on. Define each anchor once. The <duplex> wrapper is optional.

For turn-taking, put the anchor at the preceding turn's end:

<duplex>
  <A>Hello! How was your day?<e1/></A>
  <B start="e1">Pretty good. I finished the project.<e2/></B>
  <A start="e2">That's great! <laugh/></A>
</duplex>

For overlap, put the anchor where the other speaker should enter:

<duplex>
  <A>I walked to the park, <e1/>and then stopped for coffee.<e2/></A>
  <B start="e1">Mm-hmm.</B>
  <B start="e2">That sounds nice.</B>
</duplex>

Anchors express turn relationships; the model predicts the audio timing. Exact entry times are not guaranteed. Colliding turns for the same speaker follow their XML order. All references must exist, and turns must not form a cycle.

Vocal events: <laugh/>, <cough/>, <breath/>, <sigh/>, <cry/>, <sneeze/>, and <gasp/>. Write spoken backchannels as ordinary text and escape literal ampersands as &. Attributes such as type, timestamps, and duration are not supported.

Voice templates

Use your own reference audio for either speaker by passing a file path to --voice-a and --voice-b, or to voice_a and voice_b in Python:

python infer.py --model . --xml dialogue.xml \
  --voice-a speaker_a.wav --voice-b speaker_b.wav --output dialogue.wav

Use a clear, single-speaker WAV or FLAC clip between 0.5 and 60 seconds long. No transcript is needed. Audio is converted to mono and resampled automatically. You can mix a custom reference with an included voice, such as --voice-a speaker_a.wav --voice-b voice_2.

If you do not have reference audio, four example voices are included for convenience:

Template	Reference audio
voice_1	Sample 1, speaker A
voice_2	Sample 1, speaker B
voice_3	Sample 2, speaker A
voice_4	Sample 2, speaker B

License

Model weights, voice references, and conversation examples are available for noncommercial use under CC BY-NC 4.0. DuDE inference code is Apache-2.0; see LICENSE-CODE. Qwen components retain their Apache-2.0 notices in LICENSE-QWEN.

Conversation material and reference voices derive from Meta's Seamless Interaction dataset, also licensed CC BY-NC 4.0. This release includes derived transcripts, XML, speaker embeddings, and synthesized recordings. Preserve the dataset attribution when redistributing these assets.