|
Download README.md from Trelis/duplex-0: direct link, hf CLI and curl.
- Browser
- Download file 11.7 kB
-
https://huggingface.co/Trelis/duplex-0/resolve/main/README.md
- Command line
-
hf download hf://Trelis/duplex-0/README.md
-
curl -L -o README.md https://huggingface.co/Trelis/duplex-0/resolve/main/README.md
11.7 kB
| license: apache-2.0 | |
| language: | |
| - en | |
| datasets: | |
| - Trelis/duplex-pre-training-yodas3-mimi | |
| - Trelis/duplex-pre-training-podcasts-mimi | |
| - Trelis/duplex-conversational-podcasts-mimi | |
| # duplex-0 | |
| A small speech-to-speech model trained from random weights on about 750 hours of English speech. It is a research checkpoint and is far from a usable assistant: its speech is about half as intelligible as real speech on our benchmark. | |
| This page is the training report for the runs of 17 November 2026. The training scripts are in [`scripts/`](scripts/README.md). | |
| ## Training report, 17 November 2026 | |
| **Summary.** Three training options were compared on the same data, seed and settings: audio only, one text track and two text tracks. All three produce speech of similar quality: 60 to 68 confirmed words per minute after the conversational stage, against 132.8 for real speech. A text track lowers first-codebook bits per second by 12% to 14%. One text track scores highest on intelligibility after the conversational stage and audio only scores highest on turn-taking, both by margins close to the range over seeds. The text the models write is related to the speech but is far from a transcript. | |
| ### Model | |
| - 201 million parameters, or 225 million with two text tracks, which adds a second text output head. A temporal transformer of 12 layers × 768 predicts one step per 80 ms frame, and a depth transformer of 2 layers × 256 predicts that frame's audio tokens. | |
| - Audio is Mimi tokens: 8 codebooks at 12.5 frames per second from 24 kHz audio. The model has two audio tracks, one per speaker, so a frame holds 16 codebooks. | |
| - Text, when used, is the words a speaker says, written as word-piece tokens at the frame where each word starts. The number of text tokens per frame varies, and an "advance" token ends each frame's text. | |
| - **Audio only:** no text. | |
| - **One text track:** the model writes the words of the track it speaks as, just before speaking them. | |
| - **Two text tracks:** it also writes the other speaker's words, 320 ms (4 frames) after each word starts. Each text track has its own output head. | |
| ### Data | |
| - **Pre-training:** 749 hours of English speech, 408 hours from YODAS3 and 341 hours from public podcasts. | |
| - **Conversational:** 156 hours of two-speaker podcast conversation with frequent turn-taking and a separate audio track per speaker. This is a subset of the pre-training data. | |
| - Whole recordings and podcast feeds are held out for validation and testing, about 4 hours each per source. | |
| ### Schedule | |
| One H100 per option, batch 8, bf16, compiled, seed 20261117 for every run. | |
| | Stage | Length | Learning rate | | |
| | --- | --- | --- | | |
| | Pre-training | Stopped when the first codebook on held-out audio had not improved for two evaluations, one evaluation per epoch | Warmup 100 steps, then constant 6e-4 | | |
| | Conversational | One epoch, from the kept pre-training checkpoint | Peak 2e-4, warmup 60 steps, annealed over the last 20% | | |
| | Option | Kept epoch (stopped at) | Steps | Steps per second | Pre-training time | | |
| | --- | --- | --- | --- | --- | | |
| | Audio only | 10 (12) | 13,392 | 7.29 | 37 min | | |
| | One text track | 13 (15) | 20,595 | 6.25 | 63 min | | |
| | Two text tracks | 10 (12) | 17,064 | 4.57 | 73 min | | |
| The kept pre-training checkpoints are taken at a constant learning rate, so they are not annealed. The conversational epoch took 2 to 5 minutes. | |
| ### First codebook against the bigram and trigram floors | |
| The first codebook of each track carries most of the speech content. The floors are what a model scores if it only counts the previous one or two first-codebook tokens. | |
|  | |
|  | |
| Held-out pre-training audio, in bits per second (lower is better): | |
| | Option | Checkpoint | Audio | First codebook | Text, speaking track | Text, listening track | | |
| | --- | --- | --- | --- | --- | --- | | |
| | Audio only | Pre-training, epoch 10 | 817.5 | 70.7 | none | none | | |
| | One text track | Pre-training, epoch 13 | 804.1 | 62.4 | 13.0 (5.87 bits per word token) | none | | |
| | Two text tracks | Pre-training, epoch 10 | 807.7 | 60.9 | 12.3 (5.29 bits per word token) | 4.6 (1.74 bits per word token) | | |
| | Audio only | After the conversational stage | 812.5 | 70.2 | none | none | | |
| | One text track | After the conversational stage | 799.5 | 61.9 | 13.1 (5.96 bits per word token) | none | | |
| | Two text tracks | After the conversational stage | 801.4 | 60.2 | 12.2 (5.31 bits per word token) | 4.4 (1.75 bits per word token) | | |
| The bigram floor is 82.0 and the trigram floor is 80.5 on the pre-training validation set. | |
| ### Benchmarks | |
| - **Intelligibility:** the model continues a 10-second prompt for 20 seconds. The score is the number of words per minute in its speech that two transcribers agree on. Real speech scores 132.8. | |
| - **Turn-taking:** the model hears a real caller and replies. The score rewards making sound in the caller's pauses and staying quiet while the caller speaks. Real speech scores 0.47. | |
| The intelligibility prompt gives the model the audio of both speakers. A model with text tracks also gets the true words of each of its text tracks during the prompt, as in training, and writes its own text after it. Turn-taking has no prompt. Sampling is at temperature 1 with top-k 250. Figures are the mean of three seeds over 24 held-out spans, with the range over seeds in brackets. | |
|  | |
| | Option | Stage | Confirmed words per minute | Turn-taking score | Sound over the caller | | |
| | --- | --- | --- | --- | --- | | |
| | Audio only | Pre-training | 69.5 (65.6 to 74.6) | 0.37 (0.34 to 0.41) | 31% | | |
| | Audio only | Conversational | 60.4 (58.6 to 63.5) | 0.35 (0.34 to 0.38) | 17% | | |
| | One text track | Pre-training | 66.8 (61.4 to 70.1) | 0.31 (0.29 to 0.32) | 28% | | |
| | One text track | Conversational | 67.5 (61.6 to 74.4) | 0.31 (0.20 to 0.37) | 32% | | |
| | Two text tracks | Pre-training | 62.9 (61.1 to 64.8) | 0.23 (0.21 to 0.25) | 53% | | |
| | Two text tracks | Conversational | 62.4 (57.6 to 66.6) | 0.33 (0.31 to 0.37) | 33% | | |
| Real speech makes sound over the caller 7% of the time. The conversational epoch raises the turn-taking score of two text tracks from 0.23 to 0.33 and leaves its confirmed words per minute unchanged. For audio only it lowers confirmed words per minute and halves the sound over the caller. With the true prompt text, one text track is within the seed range of audio only after pre-training and above it after the conversational epoch. | |
| ### What the text tracks write | |
| After the conversational stage, the text track of the speaker the model speaks as writes these word tokens per second during the turn-taking benchmark: | |
| | Option | While the caller speaks | While the caller is quiet | | |
| | --- | --- | --- | | |
| | Real speech | 0.41 | 3.05 | | |
| | One text track | 2.35 | 5.33 | | |
| | Two text tracks | 3.53 | 7.36 | | |
| Both models write more while the caller is quiet, so they follow turns. Both still write more words than a real speaker does. | |
| How well the written text matches the speech was measured on the same runs (60 seconds generated per clip, 72 clips pooled). The model's own text is compared with a transcript of the audio it generated on that track. The caller's text, which only the two-text-track model writes, is compared with the caller's true words, so it measures speech recognition. Word error rate is substitutions, deletions and insertions divided by reference words; lower is better and 0 is a perfect match. | |
| | Text | Option | Pre-training | Conversational | Unrelated text | | |
| | --- | --- | --- | --- | --- | | |
| | Own speech | One text track | 0.90 | 0.78 | 1.34 to 1.43 | | |
| | Own speech | Two text tracks | 1.75 | 1.29 | 1.72 to 2.00 | | |
| | Caller's speech | Two text tracks | 0.84 | 0.71 | 1.18 to 1.23 | | |
| "Unrelated text" pairs each clip's written text with the reference of a different span. The transcriber used for the own-speech reference scores 0.11 to 0.18 against the true words on real audio. The written text is related to the audio and improves with the conversational stage, but it is far from a transcript: after the conversational stage the caller's text gets 45% of the caller's words right, and most errors on the model's own text are extra words. | |
| ### Samples | |
| Each clip is 30 seconds: 10 seconds of a real held-out conversation as the prompt, then 20 seconds that the model generates for both speakers (seed 1, temperature 1, top-k 250, from the checkpoint after the conversational stage). The text-track models are given the true prompt text. The "real" clip is the true continuation. Both tracks are mixed into one file, and all audio is decoded from Mimi tokens. | |
| | Prompt | Real | Audio only | One text track | Two text tracks | | |
| | --- | --- | --- | --- | --- | | |
| | 1 | <audio controls src="https://huggingface.co/Trelis/duplex-0/resolve/main/report/samples/prompt1-real.mp3"></audio> | <audio controls src="https://huggingface.co/Trelis/duplex-0/resolve/main/report/samples/prompt1-none.mp3"></audio> | <audio controls src="https://huggingface.co/Trelis/duplex-0/resolve/main/report/samples/prompt1-one.mp3"></audio> | <audio controls src="https://huggingface.co/Trelis/duplex-0/resolve/main/report/samples/prompt1-two.mp3"></audio> | | |
| | 2 | <audio controls src="https://huggingface.co/Trelis/duplex-0/resolve/main/report/samples/prompt2-real.mp3"></audio> | <audio controls src="https://huggingface.co/Trelis/duplex-0/resolve/main/report/samples/prompt2-none.mp3"></audio> | <audio controls src="https://huggingface.co/Trelis/duplex-0/resolve/main/report/samples/prompt2-one.mp3"></audio> | <audio controls src="https://huggingface.co/Trelis/duplex-0/resolve/main/report/samples/prompt2-two.mp3"></audio> | | |
| | 3 | <audio controls src="https://huggingface.co/Trelis/duplex-0/resolve/main/report/samples/prompt3-real.mp3"></audio> | <audio controls src="https://huggingface.co/Trelis/duplex-0/resolve/main/report/samples/prompt3-none.mp3"></audio> | <audio controls src="https://huggingface.co/Trelis/duplex-0/resolve/main/report/samples/prompt3-one.mp3"></audio> | <audio controls src="https://huggingface.co/Trelis/duplex-0/resolve/main/report/samples/prompt3-two.mp3"></audio> | | |
| | 4 | <audio controls src="https://huggingface.co/Trelis/duplex-0/resolve/main/report/samples/prompt4-real.mp3"></audio> | <audio controls src="https://huggingface.co/Trelis/duplex-0/resolve/main/report/samples/prompt4-none.mp3"></audio> | <audio controls src="https://huggingface.co/Trelis/duplex-0/resolve/main/report/samples/prompt4-one.mp3"></audio> | <audio controls src="https://huggingface.co/Trelis/duplex-0/resolve/main/report/samples/prompt4-two.mp3"></audio> | | |
| The files are in `report/samples/`. | |
| ### Checkpoints of this report | |
| Folders `resplit-20261117/<stage>-<option>` hold the checkpoint at the end of a stage and `resplit-20261117/<stage>-<option>-best` the best one by held-out first codebook, where `<stage>` is `pretraining` or `conversational` and `<option>` is `none`, `one` or `two`. Each folder holds `model.safetensors`, `config.json`, `checkpoint.json` and `train.json` (the full run report). The benchmark rows above use `pretraining-<option>-best` and `conversational-<option>`. | |
| ## Licence | |
| The model weights and the training scripts in `scripts/` are released under the Apache License 2.0 (`LICENSE`). | |
| The model reads and writes audio as tokens of the [Mimi](https://huggingface.co/kyutai/mimi) codec and uses the text tokenizer of [Moshi](https://huggingface.co/kyutai/moshiko-pytorch-bf16), both by Kyutai. Part of the training data comes from [YODAS3](https://huggingface.co/datasets/espnet/yodas3). | |