flux-duplex / README.md
ai-team-core's picture
Model card: flux-duplex, the system behind the TurnBench entry
31b1173
|
Raw History Blame Contribute Delete
6.22 kB
metadata
language:
  - en
tags:
  - turn-taking
  - end-of-turn
  - interruption-detection
  - voice-agents
  - full-duplex
  - audio
pipeline_tag: audio-classification

flux-duplex: a two-channel turn-taking model

Flam · November 2026 · TurnBench entry "flux-duplex"

flux-duplex is the turn-taking controller of a duplex voice agent. It listens to both sides of a conversation and predicts, 50 times a second, who will be speaking over the next two seconds. A small set of fixed rules reads that forecast and decides when a speaker's turn has ended and when a speaker is interrupting. It recognises no words and produces no speech.

Results on the TurnBench dev set

These are the 38 public dev conversations, scored with the benchmark's own scorer. Test results will be those on the leaderboard.

Task Found False alarms Median delay
End of turn 96.2% (1,832 of 1,904) 7.0% 395 ms
Interruption 99.7% (346 of 347) 7.3% 226 ms

The settings were chosen on these same conversations, so the table is in-sample. Choosing on a random half and scoring on the other half (200 splits) gives 96.2% ± 0.5 at 7.2% ± 0.8 false alarms for end of turn, and 99.5% ± 0.6 at 7.5% ± 2.1 for interruption.

The model

  • Input. Two audio channels at 16 kHz, one per speaker. Either speaker can sit in either seat.
  • Encoder. A pretrained audio encoder of about 305M parameters (12 layers), run on each channel. It gives one vector every 40 ms and looks only backwards. Its top six layers were fine-tuned; the lower six stayed frozen.
  • Turn-taking layers. 20.9M parameters on top of the encoder: a learned mix of the encoder's layers, an adapter, six transformer blocks per channel and four blocks that read both channels together, all causal, at 20 ms per frame.
  • Outputs. The main output is a forecast over 256 patterns of future speech: for each speaker, whether they talk in each of four bins covering the next 0.2, 0.6, 1.2 and 2.0 seconds. Smaller outputs give each speaker's voice activity and a few readings used by the agent but not by this entry.

The model is causal. Its output at time t depends only on audio up to t, and a frame-by-frame streaming version gives the same outputs as the offline pass to within rounding. Event times are reported at the end of the 20 ms frame in which the rule fires.

Training

  • Data. 855 hours of two-channel English conversation (1,373 conversations, 209 speakers), each speaker on their own channel, split by speaker. Noise, echo and background voices were added to the listener's channel during training.
  • Targets. Self-supervised. The forecast target is the recording's own speech activity over the next two seconds, taken from a voice-activity detector on each channel. No human turn-taking labels were used.
  • Benchmark data. No TurnBench audio or annotation, dev or test, was used in training, and the benchmark's training set was not used. The dev set was used to pick the checkpoint and the settings of the rules below.
  • Run. 24,000 steps on 20-second windows on a single GPU.

The decision rules

Three readings of the forecast are used: p_now, the chance that a speaker talks in the next 0.2 s; p_hold, the chance that a speaker who has just started keeps the floor; and each speaker's voice activity.

End of turn. While speaker S is silent, the evidence that S's turn is over is

p_now of the other speaker + 0.008 × frames the other has talked since S fell silent + 0.004 × frames of S's silence

An event is sent at the first frame, at least 0.4 s into the silence, at which the evidence reaches 1.05.

One confirmation event. If the silence is still running 2.0 s after that event and the condition still holds, one more event is sent. This is the agent looking again when nobody has taken the floor. There is at most one confirmation per silence, so a silence yields one or two events. On dev there are 8,154 end-of-turn events for 1,904 labelled turn ends. The confirmation matters because in about one labelled turn end in five the speaker's voice stops more than 0.25 s before the annotated moment, so a single early commit falls outside the scoring window. Without the confirmation the same rule finds 92.8% at the same false-alarm rate.

End of recording. One end-of-turn event per speaker is added at the end of the audio: when the call is over, every turn is over. On the single-event rule this is worth 1.7 points (91.1% to 92.8%).

Interruption. While speaker S talks, the evidence is p_hold of S + 0.008 × frames S has been talking. An event is sent at the first frame of a stretch of S's speech at which it reaches 0.5, provided the other speaker was heard in the last 2 s. There is at most one event per stretch of speech.

How the operating point was chosen

The thresholds were searched on the dev set under a false-alarm budget of 8%, and the setting with the most events found was kept. The end-of-turn thresholds were chosen with an earlier checkpoint of the same model and left as they were; the interruption thresholds were chosen with this one. The delay of the confirmation event was chosen among nine values from 0.3 to 3.0 s in the same way.

Limitations

  • It leans on the second channel. Most end-of-turn events are sent once the other speaker has begun to answer. With the other channel silenced, as for an agent that waits for its turn, the single-event rule above finds 29 to 48% of the turn ends. A separate rule for that case, which reads the forecast for the speaker alone, is used in our live agent and is not part of this entry.
  • Sound only. The model does not know what was said, so turn ends that depend on meaning are out of its reach.
  • English, two clean channels. Nothing else was measured.
  • The dev figures are in-sample for the checkpoint and the settings, as stated above.

Availability

The model is internal to Flam and its weights are not published. This page is the description of the system behind the TurnBench entry "flux-duplex". The entry itself is the list of event times the system produced on the benchmark audio, as the benchmark asks for.