Turn-Detection-Jev

Audio turn completion, interruptions, and backchannels

Fine-tuned Qwen3-Omni-30B-A3B-Instruct · Five semantic actions / seven state labels · Apache-2.0

GitHub code · Model weights · Quick start · License

Language: English | 中文

Turn-Detection-Jev predicts when a speech assistant should consider responding, waiting, or handling a user interruption. It takes audio and the application's current assistant playback state, then scores the labels available in that state. It is fine-tuned from Qwen3-Omni-30B-A3B-Instruct.

Contents

Core features

Capability Description
Turn completion Distinguishes finished speech, incomplete speech, and background when the assistant is silent
Interruption and backchannel Distinguishes completed interruptions, brief feedback, unfinished speech, and background during playback
State-conditioned scoring Uses three listening labels or four speaking labels according to caller-supplied playback state
Single-forward prediction Uses Qwen3-Omni-30B-A3B-Instruct audio understanding and reads candidate answer-token logits in one forward pass, without generating explanations
Application integration The GitHub runtime provides CLI inference and an HTTP API returning labels and probabilities
Open weights Apache-2.0, retaining attribution to the Qwen3-Omni-30B-A3B-Instruct base model

Labels

Assistant playback Output label Meaning Illustrative example (not an observed prediction)
Off listen_complete The user has finished and yielded the floor “Check tomorrow's weather in Beijing.”
Off listen_incomplete The user is still speaking or holding the floor “I'd like to ask about, um…”
Off listen_background Background unrelated to the target interaction Ambient noise, bystanders, or playback echo
On speak_interrupt A completed request to stop, correct, or take the floor “Stop a moment. I meant tomorrow.”
On speak_backchannel Finished brief feedback without requesting the floor “Mm-hmm, right.”
On speak_incomplete The user has started but not finished an utterance “Wait, what I wanted was…”
On speak_background Background unrelated to the target interaction Ambient noise, bystanders, or playback echo

The application must provide the real assistant_speaking state. The same words can express different intentions in context; these examples are illustrations, not measured model predictions.

Benchmark results

Public evaluation sets

Turn-Detection-Jev scores come from the complete native Transformers run merged-20261111_051726-1672084 on 2026-11-11, using temperature T=1 with calibration disabled. Other models retain their historical results.

Dataset Samples Correct Accuracy Semantic Macro-F1
Easy-Turn 800 789 98.625% 99.00%
Smart-Test-4000 4,000 3,871 96.775% 96.81%

Evaluation conditions: These are state-conditioned single-window results. Playback state was constructed using label rules and sample-ID hashes. Input audio was mono 16 kHz PCM16, retaining at most the final 8 seconds. Macro averages cover ground-truth-supported labels only. Easy-Turn covers five state labels and Smart-Test three; neither has background ground truth.

Comparison on shared samples

The main comparison uses original semantic labels: listen_complete→complete; listen_incomplete/speak_incomplete→incomplete; speak_interrupt→wait; speak_backchannel→backchannel; both background labels→invalid. Only labels are merged; predictions are retained, and options are not renormalized based on ground truth.

Metric comparison

Dataset Model N Accuracy Macro precision Macro recall Macro-F1
easyturn TurnSense 1.1 GPU 600 96.17% 96.19% 96.17% 96.17%
easyturn Smart-Turn 3.2 CPU 600 76.00% 78.21% 76.00% 75.52%
easyturn Smart-Turn 3.2 GPU 600 77.17% 78.26% 77.17% 76.94%
easyturn Qwen3-Omni-30B-A3B-Instruct 600 92.33% 93.40% 92.33% 92.53%
easyturn Turn-Detection-Jev 600 98.17% 98.33% 98.17% 98.25%
smart-test TurnSense 1.1 GPU 4000 26.28% 75.97% 26.28% 38.86%
smart-test Smart-Turn 3.2 CPU 4000 92.50% 92.59% 92.50% 92.50%
smart-test Smart-Turn 3.2 GPU 4000 93.88% 93.89% 93.88% 93.87%
smart-test Qwen3-Omni-30B-A3B-Instruct 4000 69.40% 80.43% 69.40% 66.35%
smart-test Turn-Detection-Jev 4000 96.78% 96.87% 96.78% 96.81%

Metrics on shared evaluation samples

Accuracy is computed over all shared samples. Precision, recall, and F1 are equally averaged over the two ground-truth target classes, complete and incomplete. Predictions outside those targets remain errors and are not renormalized.

Turn-Detection-Jev received constructed assistant-state metadata that the other audio models did not receive. Services and execution environments also differ. These figures compare the current Jev run with historical baselines; they are not a controlled comparison with identical information. The TurnSense figures here come from our evaluation, not its official model card.

Client request latency

Dataset Model N Mean ms P50 ms P90 ms Serial samples/s
easyturn TurnSense 1.1 GPU 600 638.60 648.29 657.59 1.57
easyturn Smart-Turn 3.2 CPU 600 64.91 47.66 54.51 15.41
easyturn Smart-Turn 3.2 GPU 600 5.36 4.18 5.26 186.68
easyturn Qwen3-Omni-30B-A3B-Instruct 600 222.09 226.87 247.68 4.50
easyturn Turn-Detection-Jev 600 120.38 94.68 160.56 8.31
smart-test TurnSense 1.1 GPU 4000 543.53 649.01 659.98 1.84
smart-test Smart-Turn 3.2 CPU 4000 165.56 192.63 296.12 6.04
smart-test Smart-Turn 3.2 GPU 4000 4.07 3.42 4.63 245.65
smart-test Qwen3-Omni-30B-A3B-Instruct 4000 229.76 221.62 275.89 4.35
smart-test Turn-Detection-Jev 4000 97.50 96.29 110.90 10.26

Client request latency and serial throughput

Turn-Detection-Jev latency comes from the native Transformers run above and includes local audio reading/decoding, preprocessing and inference. First-call overhead is retained; independent GPU kernel time was not measured. Other models retain historical timing.

The chart uses logarithmic axes. Historical ONNX runs used warmup=20 and remote runs warmup=0. ONNX timing includes mixing, resampling, features and inference; remote timing additionally includes encoding, HTTP, queueing and parsing. Both historical paths exclude file reading/decoding, whereas this Jev run includes it. Hardware, models and execution environments also differ, so this is not a controlled speed ranking.

Table samples/s equals N × 1000 / cumulative per-sample latency; it is neither whole-run wall-clock throughput nor concurrent capacity. Independent gpu_latency_ms and inference_latency_ms were not recorded in this run.

Quick start

This Hugging Face repository contains model artifacts and documentation. Inference, serving, evaluation tools, and test-set instructions are maintained in the linked GitHub repository.

Copy the complete project to the server, or obtain its source first (Git does not include the test WAV files):

git clone https://github.com/YOUR_GITHUB_ORG/Turn-Detection-Jev.git
cd Turn-Detection-Jev
git checkout main

1. Install dependencies

Activate an isolated Python >= 3.10 environment and install CUDA-enabled PyTorch suitable for the server first. From the project root, run:

bash scripts/01_install.sh

Alternatively, run the equivalent commands directly (choose either the script or these commands):

python -m pip install -e '.[datasets]'

This runs python -m pip install -e '.[datasets]' and prints the PyTorch version and CUDA availability. Dependencies are declared in pyproject.toml; the datasets extra installs PyArrow for raw Parquet conversion in step 2; merged weights do not require the PEFT/adapter extra. Confirm CUDA available: True before continuing. All three scripts use the current environment's python, falling back to python3, or an explicit PYTHON=/absolute/path/to/python. Use the same environment for all steps.

2. Convert and validate the original test sets

Before step 2, keep this layout in the code repository when using default paths:

Turn-Detection-Jev/
├── datasets/
│   ├── README.md                         # Dataset instructions; recommended
│   ├── Easy-Turn-Testset/                 # Complete original Easy-Turn test set
│   ├── smart-turn-data-v3.2-test/         # Complete original Smart-Turn v3.2 test set
│   ├── easyturn/                         # Documentation/licenses copied during export
│   │   ├── LICENSE
│   │   ├── README.md
│   │   └── SOURCE_README.md
│   ├── smart-test-4000/                   # Documentation/licenses copied during export
│   │   ├── LICENSE
│   │   ├── README.md
│   │   └── SOURCE_README.md
│   └── v3-public/                        # Generated by step 2; do not create an empty directory
│       ├── easyturn/                     # audio/, test.jsonl, summary.json, docs/licenses
│       └── smart-test-4000/              # audio/, test.jsonl, summary.json, docs/licenses
└── results/
    └── reference/                        # Frozen IDs, labels, states and audio hashes
        ├── easyturn.predictions.jsonl
        └── smart-test-4000.predictions.jsonl

Keep the original upstream layouts and audio. See the code repository's datasets/README.md for required lists and Parquet shards. Old audio/, test.jsonl and summary.json files directly under datasets/easyturn/ and datasets/smart-test-4000/ are not inputs to the current steps 2 and 3 and do not need uploading; retain the three documentation/license files shown in each directory.

Step 3 reads the generated datasets/v3-public/; keep its audio, manifests and accompanying files together. Step 2 validates and reuses a complete existing export without requiring the original datasets. Rebuilding requires the originals, documentation/licenses and results/reference/ again. Keep the remaining source code and configuration files as usual.

With the default layout above, run:

bash scripts/02_validate_data.sh

Use the original datasets on the evaluation server (paths may contain spaces):

EASYTURN_DIR=/path/to/Easy-Turn-Testset \
SMART_TEST_DIR=/path/to/smart-turn-data-v3.2-test \
  bash scripts/02_validate_data.sh

Without these variables, sources default to datasets/Easy-Turn-Testset and datasets/smart-turn-data-v3.2-test inside this project. The default input is the full original Smart-Turn v3.2 test snapshot; conversion selects the frozen historical 4000 IDs. You can also point SMART_TEST_DIR to an existing 4000-row raw Parquet subset.

Step 2 converts to datasets/v3-public/ and validates paths, SHA256 hashes, audio format and label/state consistency for 800 Easy-Turn and 4000 Smart-Test samples. Audio is mono 16 kHz PCM16 WAV, keeping at most the last 8 seconds. No model is loaded and sources are unchanged. Later runs fully validate and reuse existing output. Damaged or incomplete output causes an error; use a new DATASETS_DIR to rebuild.

Git excludes datasets/**/audio/. Convert from the originals on the server without uploading a local audio export. Step 2 does not download the originals automatically.

To change the output directory, set the same DATASETS_DIR for steps 2 and 3:

EASYTURN_DIR=/path/to/Easy-Turn-Testset \
SMART_TEST_DIR=/path/to/smart-turn-data-v3.2-test \
DATASETS_DIR=datasets/v3-public-new bash scripts/02_validate_data.sh
DATASETS_DIR=datasets/v3-public-new bash scripts/03_evaluate.sh

Equivalent direct commands (choose either these or the script):

python scripts/prepare_v3_testsets.py \
  --easyturn /path/to/Easy-Turn-Testset \
  --smart-test /path/to/smart-turn-data-v3.2-test \
  --output datasets/v3-public --reuse-existing
python evaluate.py --manifest datasets/v3-public/easyturn/test.jsonl --validate-only
python evaluate.py --manifest datasets/v3-public/smart-test-4000/test.jsonl --validate-only

3. Evaluate merged weights

Place the full model in models/Turn-Detection-Jev/. Include config.json, all weight shards and their index, processor/tokenizer files and the original scoring_head.json. Do not leave an adapter's adapter_config.json in this directory: it selects the adapter loading path.

bash scripts/03_evaluate.sh

# Optional: specify a model directory and a new output directory.
# Relative paths are resolved from the project root.
bash scripts/03_evaluate.sh models/Turn-Detection-Jev results/runs/merged-check

Alternatively, run the equivalent commands directly (choose either the script or these commands):

RUN_OUTPUT="results/runs/merged-$(date +%Y%m%d_%H%M%S)-$$"
python evaluate.py --model models/Turn-Detection-Jev \
  --device cuda:0 --dtype bfloat16 --attn-implementation sdpa \
  --calibration hf/calibration.json \
  --manifest datasets/v3-public/easyturn/test.jsonl --out "$RUN_OUTPUT/easyturn" \
  --manifest datasets/v3-public/smart-test-4000/test.jsonl --out "$RUN_OUTPUT/smart-test-4000"

By default this reads the converted data from datasets/v3-public/ validated in step 2. This starts one evaluate.py process, loads the model once and sequentially evaluates all 800 Easy-Turn and 4000 Smart-Test samples, without --limit. It uses native Transformers/PyTorch, one GPU (cuda:0), BF16, SDPA and temperature calibration from hf/calibration.json. No HTTP server, vLLM, separate base-model download or LoRA weights are needed. Select another GPU with DEVICE=cuda:1 bash scripts/03_evaluate.sh; the selected GPU must have enough memory for inference.

The script now loads hf/calibration.json by default: listening uses choice:3 (T≈1.01754) and speaking uses choice:4 (T≈1.37385). The report records provenance.calibration_enabled=true. To select another file:

CALIBRATION=/path/to/calibration.json bash scripts/03_evaluate.sh /path/to/model

The published 2026-11-11 scores above came from a run with calibration disabled and have not been replaced with a calibrated rerun. Use this command to match that T=1 setting:

CALIBRATION=none bash scripts/03_evaluate.sh /path/to/model

Temperature scaling adjusts probabilities and confidence. Positive temperature preserves label ordering mathematically and does not guarantee an improvement in Accuracy/F1.

Each run defaults to a new results/runs/merged-TIMESTAMP-PID/ directory with per-dataset predictions.jsonl and report.json files. Inspect classification.accuracy, classification.macro_f1, semantic.macro_f1 and is_full_run=true. The main benchmark table reports semantic Macro-F1. A specified output directory must not already exist; scripts stop on errors.

These are single-window results conditioned on constructed assistant_speaking state. Published Jev scores now use the complete native Transformers run above; historical ms-swift results remain in the code repository under results/reference/.

Training and evaluation

The model was fine-tuned with LoRA rank 32, alpha 64, learning rate 5e-5, and a three-epoch budget. The processed pool contains 156,925 samples from Easy-Turn, Smart-Turn and MUSAN noise, with 141,232 assigned to training. On the separate internal test set of 3,923 samples, accuracy was 97.04% and seven-label Macro-F1 was 97.87%.

See EVALUATION.md for data distributions, label recalls, experiment provenance and timing boundaries.

Limitations

  • Public sets do not assess background recall. Smart-Test's speaking group contains incomplete speech only, so its 99.90% accuracy does not establish interruption/backchannel discrimination.
  • Performance is conditional on correct external playback state. These results do not establish streaming event latency or robustness to deployment devices, echo, noise and accents.
  • This release requires a Qwen3-Omni-30B-A3B-Instruct GPU environment. CPU, ONNX and quantized deployments have not been benchmarked.

License and acknowledgements

Fine-tuned weights are released under Apache License 2.0, permitting no-charge research, personal and commercial use, modification and redistribution subject to its terms.

The base model was developed and released by the Qwen team: Qwen3-Omni-30B-A3B-Instruct. Its official model page declares Apache-2.0; the official license text is available in Qwen3-Omni/LICENSE. Base-model attribution and fine-tuning modifications are recorded in NOTICE.

Thanks to Qwen, ms-swift, OmniJev/OneJev, Easy-Turn, Smart-Turn and MUSAN. Datasets and dependencies retain their own licenses. This independent derivative is not endorsed by the upstream authors.

Source code and feedback

GitHub source · GitHub Issues

Downloads last month
27
Safetensors
Model size
32B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for Beagledata/Turn-Detection-Jev

Finetuned
(45)
this model

Datasets used to train Beagledata/Turn-Detection-Jev