|
Download README.md from Flam-ai/flux-duplex: direct link, hf CLI and curl.
- Browser
- Download file 6.22 kB
-
https://huggingface.co/Flam-ai/flux-duplex/resolve/main/README.md
- Command line
-
hf download hf://Flam-ai/flux-duplex/README.md
-
curl -L -o README.md https://huggingface.co/Flam-ai/flux-duplex/resolve/main/README.md
6.22 kB
| language: | |
| - en | |
| tags: | |
| - turn-taking | |
| - end-of-turn | |
| - interruption-detection | |
| - voice-agents | |
| - full-duplex | |
| - audio | |
| pipeline_tag: audio-classification | |
| # flux-duplex: a two-channel turn-taking model | |
| Flam · November 2026 · TurnBench entry "flux-duplex" | |
| flux-duplex is the turn-taking controller of a duplex voice agent. It listens to both sides of a conversation and | |
| predicts, 50 times a second, who will be speaking over the next two seconds. A small set of fixed rules reads that | |
| forecast and decides when a speaker's turn has ended and when a speaker is interrupting. It recognises no words and | |
| produces no speech. | |
| ## Results on the TurnBench dev set | |
| These are the 38 public dev conversations, scored with the benchmark's own scorer. Test results will be those on the | |
| leaderboard. | |
| | Task | Found | False alarms | Median delay | | |
| | --- | --- | --- | --- | | |
| | End of turn | 96.2% (1,832 of 1,904) | 7.0% | 395 ms | | |
| | Interruption | 99.7% (346 of 347) | 7.3% | 226 ms | | |
| The settings were chosen on these same conversations, so the table is in-sample. Choosing on a random half and scoring | |
| on the other half (200 splits) gives 96.2% ± 0.5 at 7.2% ± 0.8 false alarms for end of turn, and 99.5% ± 0.6 at | |
| 7.5% ± 2.1 for interruption. | |
| ## The model | |
| - **Input.** Two audio channels at 16 kHz, one per speaker. Either speaker can sit in either seat. | |
| - **Encoder.** A pretrained audio encoder of about 305M parameters (12 layers), run on each channel. It gives one | |
| vector every 40 ms and looks only backwards. Its top six layers were fine-tuned; the lower six stayed frozen. | |
| - **Turn-taking layers.** 20.9M parameters on top of the encoder: a learned mix of the encoder's layers, an adapter, | |
| six transformer blocks per channel and four blocks that read both channels together, all causal, at 20 ms per frame. | |
| - **Outputs.** The main output is a forecast over 256 patterns of future speech: for each speaker, whether they talk in | |
| each of four bins covering the next 0.2, 0.6, 1.2 and 2.0 seconds. Smaller outputs give each speaker's voice activity | |
| and a few readings used by the agent but not by this entry. | |
| The model is causal. Its output at time t depends only on audio up to t, and a frame-by-frame streaming version gives | |
| the same outputs as the offline pass to within rounding. Event times are reported at the end of the 20 ms frame in which the rule fires. | |
| ## Training | |
| - **Data.** 855 hours of two-channel English conversation (1,373 conversations, 209 speakers), each speaker on their own | |
| channel, split by speaker. Noise, echo and background voices were added to the listener's channel during training. | |
| - **Targets.** Self-supervised. The forecast target is the recording's own speech activity over the next two seconds, | |
| taken from a voice-activity detector on each channel. No human turn-taking labels were used. | |
| - **Benchmark data.** No TurnBench audio or annotation, dev or test, was used in training, and the benchmark's training | |
| set was not used. The dev set was used to pick the checkpoint and the settings of the rules below. | |
| - **Run.** 24,000 steps on 20-second windows on a single GPU. | |
| ## The decision rules | |
| Three readings of the forecast are used: `p_now`, the chance that a speaker talks in the next 0.2 s; `p_hold`, the | |
| chance that a speaker who has just started keeps the floor; and each speaker's voice activity. | |
| **End of turn.** While speaker S is silent, the evidence that S's turn is over is | |
| p_now of the other speaker + 0.008 × frames the other has talked since S fell silent + 0.004 × frames of S's silence | |
| An event is sent at the first frame, at least 0.4 s into the silence, at which the evidence reaches 1.05. | |
| **One confirmation event.** If the silence is still running 2.0 s after that event and the condition still holds, one | |
| more event is sent. This is the agent looking again when nobody has taken the floor. There is at most one confirmation | |
| per silence, so a silence yields one or two events. On dev there are 8,154 end-of-turn events for 1,904 labelled turn | |
| ends. The confirmation matters because in about one labelled turn end in five the speaker's voice stops more than | |
| 0.25 s before the annotated moment, so a single early commit falls outside the scoring window. Without the | |
| confirmation the same rule finds 92.8% at the same false-alarm rate. | |
| **End of recording.** One end-of-turn event per speaker is added at the end of the audio: when the call is over, every | |
| turn is over. On the single-event rule this is worth 1.7 points (91.1% to 92.8%). | |
| **Interruption.** While speaker S talks, the evidence is `p_hold of S + 0.008 × frames S has been talking`. An event is | |
| sent at the first frame of a stretch of S's speech at which it reaches 0.5, provided the other speaker was heard in | |
| the last 2 s. There is at most one event per stretch of speech. | |
| ## How the operating point was chosen | |
| The thresholds were searched on the dev set under a false-alarm budget of 8%, and the setting with the most events | |
| found was kept. The end-of-turn thresholds were chosen with an earlier checkpoint of the same model and left as they | |
| were; the interruption thresholds were chosen with this one. The delay of the confirmation event was chosen among nine | |
| values from 0.3 to 3.0 s in the same way. | |
| ## Limitations | |
| - **It leans on the second channel.** Most end-of-turn events are sent once the other speaker has begun to answer. With | |
| the other channel silenced, as for an agent that waits for its turn, the single-event rule above finds 29 to 48% of | |
| the turn ends. A separate rule for that case, which reads the forecast for the speaker alone, is used in our live | |
| agent and is not part of this entry. | |
| - **Sound only.** The model does not know what was said, so turn ends that depend on meaning are out of its reach. | |
| - **English, two clean channels.** Nothing else was measured. | |
| - **The dev figures are in-sample** for the checkpoint and the settings, as stated above. | |
| ## Availability | |
| The model is internal to Flam and its weights are not published. This page is the description of the system behind | |
| the TurnBench entry "flux-duplex". The entry itself is the list of event times the system produced on the benchmark | |
| audio, as the benchmark asks for. | |