Instructions to use Beagledata/Turn-Detection-Jev with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Beagledata/Turn-Detection-Jev with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("audio-classification", model="Beagledata/Turn-Detection-Jev")# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("Beagledata/Turn-Detection-Jev") model = AutoModelForMultimodalLM.from_pretrained("Beagledata/Turn-Detection-Jev", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Turn-Detection-Jev
Audio turn completion, interruptions, and backchannels
Fine-tuned Qwen3-Omni-30B-A3B-Instruct · Five semantic actions / seven state labels · Apache-2.0
GitHub code · Model weights · Quick start · License
Language: English | 中文
Turn-Detection-Jev predicts when a speech assistant should consider responding, waiting, or handling a user interruption. It takes audio and the application's current assistant playback state, then scores the labels available in that state. It is fine-tuned from Qwen3-Omni-30B-A3B-Instruct.
Contents
- Core features
- Labels
- Benchmark results
- Metric comparison
- Client request latency
- Quick start
- Training and evaluation
- Limitations
- License and acknowledgements
- Source code and feedback
Core features
| Capability | Description |
|---|---|
| Turn completion | Distinguishes finished speech, incomplete speech, and background when the assistant is silent |
| Interruption and backchannel | Distinguishes completed interruptions, brief feedback, unfinished speech, and background during playback |
| State-conditioned scoring | Uses three listening labels or four speaking labels according to caller-supplied playback state |
| Single-forward prediction | Uses Qwen3-Omni-30B-A3B-Instruct audio understanding and reads candidate answer-token logits in one forward pass, without generating explanations |
| Application integration | The GitHub runtime provides CLI inference and an HTTP API returning labels and probabilities |
| Open weights | Apache-2.0, retaining attribution to the Qwen3-Omni-30B-A3B-Instruct base model |
Labels
| Assistant playback | Output label | Meaning | Illustrative example (not an observed prediction) |
|---|---|---|---|
| Off | listen_complete |
The user has finished and yielded the floor | “Check tomorrow's weather in Beijing.” |
| Off | listen_incomplete |
The user is still speaking or holding the floor | “I'd like to ask about, um…” |
| Off | listen_background |
Background unrelated to the target interaction | Ambient noise, bystanders, or playback echo |
| On | speak_interrupt |
A completed request to stop, correct, or take the floor | “Stop a moment. I meant tomorrow.” |
| On | speak_backchannel |
Finished brief feedback without requesting the floor | “Mm-hmm, right.” |
| On | speak_incomplete |
The user has started but not finished an utterance | “Wait, what I wanted was…” |
| On | speak_background |
Background unrelated to the target interaction | Ambient noise, bystanders, or playback echo |
The application must provide the real assistant_speaking state. The same words can express different intentions in context; these examples are illustrations, not measured model predictions.
Benchmark results
Public evaluation sets
Turn-Detection-Jev scores come from the complete native Transformers run merged-20261111_051726-1672084 on 2026-11-11, using temperature T=1 with calibration disabled. Other models retain their historical results.
| Dataset | Samples | Correct | Accuracy | Semantic Macro-F1 |
|---|---|---|---|---|
| Easy-Turn | 800 | 789 | 98.625% | 99.00% |
| Smart-Test-4000 | 4,000 | 3,871 | 96.775% | 96.81% |
Evaluation conditions: These are state-conditioned single-window results. Playback state was constructed using label rules and sample-ID hashes. Input audio was mono 16 kHz PCM16, retaining at most the final 8 seconds. Macro averages cover ground-truth-supported labels only. Easy-Turn covers five state labels and Smart-Test three; neither has background ground truth.
Comparison on shared samples
The main comparison uses original semantic labels: listen_complete→complete; listen_incomplete/speak_incomplete→incomplete; speak_interrupt→wait; speak_backchannel→backchannel; both background labels→invalid. Only labels are merged; predictions are retained, and options are not renormalized based on ground truth.
Metric comparison
| Dataset | Model | N | Accuracy | Macro precision | Macro recall | Macro-F1 |
|---|---|---|---|---|---|---|
| easyturn | TurnSense 1.1 GPU | 600 | 96.17% | 96.19% | 96.17% | 96.17% |
| easyturn | Smart-Turn 3.2 CPU | 600 | 76.00% | 78.21% | 76.00% | 75.52% |
| easyturn | Smart-Turn 3.2 GPU | 600 | 77.17% | 78.26% | 77.17% | 76.94% |
| easyturn | Qwen3-Omni-30B-A3B-Instruct | 600 | 92.33% | 93.40% | 92.33% | 92.53% |
| easyturn | Turn-Detection-Jev | 600 | 98.17% | 98.33% | 98.17% | 98.25% |
| smart-test | TurnSense 1.1 GPU | 4000 | 26.28% | 75.97% | 26.28% | 38.86% |
| smart-test | Smart-Turn 3.2 CPU | 4000 | 92.50% | 92.59% | 92.50% | 92.50% |
| smart-test | Smart-Turn 3.2 GPU | 4000 | 93.88% | 93.89% | 93.88% | 93.87% |
| smart-test | Qwen3-Omni-30B-A3B-Instruct | 4000 | 69.40% | 80.43% | 69.40% | 66.35% |
| smart-test | Turn-Detection-Jev | 4000 | 96.78% | 96.87% | 96.78% | 96.81% |
Accuracy is computed over all shared samples. Precision, recall, and F1 are equally averaged over the two ground-truth target classes, complete and incomplete. Predictions outside those targets remain errors and are not renormalized.
Turn-Detection-Jev received constructed assistant-state metadata that the other audio models did not receive. Services and execution environments also differ. These figures compare the current Jev run with historical baselines; they are not a controlled comparison with identical information. The TurnSense figures here come from our evaluation, not its official model card.
Client request latency
| Dataset | Model | N | Mean ms | P50 ms | P90 ms | Serial samples/s |
|---|---|---|---|---|---|---|
| easyturn | TurnSense 1.1 GPU | 600 | 638.60 | 648.29 | 657.59 | 1.57 |
| easyturn | Smart-Turn 3.2 CPU | 600 | 64.91 | 47.66 | 54.51 | 15.41 |
| easyturn | Smart-Turn 3.2 GPU | 600 | 5.36 | 4.18 | 5.26 | 186.68 |
| easyturn | Qwen3-Omni-30B-A3B-Instruct | 600 | 222.09 | 226.87 | 247.68 | 4.50 |
| easyturn | Turn-Detection-Jev | 600 | 120.38 | 94.68 | 160.56 | 8.31 |
| smart-test | TurnSense 1.1 GPU | 4000 | 543.53 | 649.01 | 659.98 | 1.84 |
| smart-test | Smart-Turn 3.2 CPU | 4000 | 165.56 | 192.63 | 296.12 | 6.04 |
| smart-test | Smart-Turn 3.2 GPU | 4000 | 4.07 | 3.42 | 4.63 | 245.65 |
| smart-test | Qwen3-Omni-30B-A3B-Instruct | 4000 | 229.76 | 221.62 | 275.89 | 4.35 |
| smart-test | Turn-Detection-Jev | 4000 | 97.50 | 96.29 | 110.90 | 10.26 |
Turn-Detection-Jev latency comes from the native Transformers run above and includes local audio reading/decoding, preprocessing and inference. First-call overhead is retained; independent GPU kernel time was not measured. Other models retain historical timing.
The chart uses logarithmic axes. Historical ONNX runs used warmup=20 and remote runs warmup=0. ONNX timing includes mixing, resampling, features and inference; remote timing additionally includes encoding, HTTP, queueing and parsing. Both historical paths exclude file reading/decoding, whereas this Jev run includes it. Hardware, models and execution environments also differ, so this is not a controlled speed ranking.
Table samples/s equals N × 1000 / cumulative per-sample latency; it is neither whole-run wall-clock throughput nor concurrent capacity. Independent gpu_latency_ms and inference_latency_ms were not recorded in this run.
Quick start
This Hugging Face repository contains model artifacts and documentation. Inference, serving, evaluation tools, and test-set instructions are maintained in the linked GitHub repository.
Copy the complete project to the server, or obtain its source first (Git does not include the test WAV files):
git clone https://github.com/YOUR_GITHUB_ORG/Turn-Detection-Jev.git
cd Turn-Detection-Jev
git checkout main
1. Install dependencies
Activate an isolated Python >= 3.10 environment and install CUDA-enabled PyTorch suitable for the server first. From the project root, run:
bash scripts/01_install.sh
Alternatively, run the equivalent commands directly (choose either the script or these commands):
python -m pip install -e '.[datasets]'
This runs python -m pip install -e '.[datasets]' and prints the PyTorch version and CUDA availability. Dependencies are declared in pyproject.toml; the datasets extra installs PyArrow for raw Parquet conversion in step 2; merged weights do not require the PEFT/adapter extra. Confirm CUDA available: True before continuing. All three scripts use the current environment's python, falling back to python3, or an explicit PYTHON=/absolute/path/to/python. Use the same environment for all steps.
2. Convert and validate the original test sets
Before step 2, keep this layout in the code repository when using default paths:
Turn-Detection-Jev/
├── datasets/
│ ├── README.md # Dataset instructions; recommended
│ ├── Easy-Turn-Testset/ # Complete original Easy-Turn test set
│ ├── smart-turn-data-v3.2-test/ # Complete original Smart-Turn v3.2 test set
│ ├── easyturn/ # Documentation/licenses copied during export
│ │ ├── LICENSE
│ │ ├── README.md
│ │ └── SOURCE_README.md
│ ├── smart-test-4000/ # Documentation/licenses copied during export
│ │ ├── LICENSE
│ │ ├── README.md
│ │ └── SOURCE_README.md
│ └── v3-public/ # Generated by step 2; do not create an empty directory
│ ├── easyturn/ # audio/, test.jsonl, summary.json, docs/licenses
│ └── smart-test-4000/ # audio/, test.jsonl, summary.json, docs/licenses
└── results/
└── reference/ # Frozen IDs, labels, states and audio hashes
├── easyturn.predictions.jsonl
└── smart-test-4000.predictions.jsonl
Keep the original upstream layouts and audio. See the code repository's datasets/README.md for required lists and Parquet shards. Old audio/, test.jsonl and summary.json files directly under datasets/easyturn/ and datasets/smart-test-4000/ are not inputs to the current steps 2 and 3 and do not need uploading; retain the three documentation/license files shown in each directory.
Step 3 reads the generated datasets/v3-public/; keep its audio, manifests and accompanying files together. Step 2 validates and reuses a complete existing export without requiring the original datasets. Rebuilding requires the originals, documentation/licenses and results/reference/ again. Keep the remaining source code and configuration files as usual.
With the default layout above, run:
bash scripts/02_validate_data.sh
Use the original datasets on the evaluation server (paths may contain spaces):
EASYTURN_DIR=/path/to/Easy-Turn-Testset \
SMART_TEST_DIR=/path/to/smart-turn-data-v3.2-test \
bash scripts/02_validate_data.sh
Without these variables, sources default to datasets/Easy-Turn-Testset and datasets/smart-turn-data-v3.2-test inside this project. The default input is the full original Smart-Turn v3.2 test snapshot; conversion selects the frozen historical 4000 IDs. You can also point SMART_TEST_DIR to an existing 4000-row raw Parquet subset.
Step 2 converts to datasets/v3-public/ and validates paths, SHA256 hashes, audio format and label/state consistency for 800 Easy-Turn and 4000 Smart-Test samples. Audio is mono 16 kHz PCM16 WAV, keeping at most the last 8 seconds. No model is loaded and sources are unchanged. Later runs fully validate and reuse existing output. Damaged or incomplete output causes an error; use a new DATASETS_DIR to rebuild.
Git excludes datasets/**/audio/. Convert from the originals on the server without uploading a local audio export. Step 2 does not download the originals automatically.
To change the output directory, set the same DATASETS_DIR for steps 2 and 3:
EASYTURN_DIR=/path/to/Easy-Turn-Testset \
SMART_TEST_DIR=/path/to/smart-turn-data-v3.2-test \
DATASETS_DIR=datasets/v3-public-new bash scripts/02_validate_data.sh
DATASETS_DIR=datasets/v3-public-new bash scripts/03_evaluate.sh
Equivalent direct commands (choose either these or the script):
python scripts/prepare_v3_testsets.py \
--easyturn /path/to/Easy-Turn-Testset \
--smart-test /path/to/smart-turn-data-v3.2-test \
--output datasets/v3-public --reuse-existing
python evaluate.py --manifest datasets/v3-public/easyturn/test.jsonl --validate-only
python evaluate.py --manifest datasets/v3-public/smart-test-4000/test.jsonl --validate-only
3. Evaluate merged weights
Place the full model in models/Turn-Detection-Jev/. Include config.json, all weight shards and their index, processor/tokenizer files and the original scoring_head.json. Do not leave an adapter's adapter_config.json in this directory: it selects the adapter loading path.
bash scripts/03_evaluate.sh
# Optional: specify a model directory and a new output directory.
# Relative paths are resolved from the project root.
bash scripts/03_evaluate.sh models/Turn-Detection-Jev results/runs/merged-check
Alternatively, run the equivalent commands directly (choose either the script or these commands):
RUN_OUTPUT="results/runs/merged-$(date +%Y%m%d_%H%M%S)-$$"
python evaluate.py --model models/Turn-Detection-Jev \
--device cuda:0 --dtype bfloat16 --attn-implementation sdpa \
--calibration hf/calibration.json \
--manifest datasets/v3-public/easyturn/test.jsonl --out "$RUN_OUTPUT/easyturn" \
--manifest datasets/v3-public/smart-test-4000/test.jsonl --out "$RUN_OUTPUT/smart-test-4000"
By default this reads the converted data from datasets/v3-public/ validated in step 2. This starts one evaluate.py process, loads the model once and sequentially evaluates all 800 Easy-Turn and 4000 Smart-Test samples, without --limit. It uses native Transformers/PyTorch, one GPU (cuda:0), BF16, SDPA and temperature calibration from hf/calibration.json. No HTTP server, vLLM, separate base-model download or LoRA weights are needed. Select another GPU with DEVICE=cuda:1 bash scripts/03_evaluate.sh; the selected GPU must have enough memory for inference.
The script now loads hf/calibration.json by default: listening uses choice:3 (T≈1.01754) and speaking uses choice:4 (T≈1.37385). The report records provenance.calibration_enabled=true. To select another file:
CALIBRATION=/path/to/calibration.json bash scripts/03_evaluate.sh /path/to/model
The published 2026-11-11 scores above came from a run with calibration disabled and have not been replaced with a calibrated rerun. Use this command to match that T=1 setting:
CALIBRATION=none bash scripts/03_evaluate.sh /path/to/model
Temperature scaling adjusts probabilities and confidence. Positive temperature preserves label ordering mathematically and does not guarantee an improvement in Accuracy/F1.
Each run defaults to a new results/runs/merged-TIMESTAMP-PID/ directory with per-dataset predictions.jsonl and report.json files. Inspect classification.accuracy, classification.macro_f1, semantic.macro_f1 and is_full_run=true. The main benchmark table reports semantic Macro-F1. A specified output directory must not already exist; scripts stop on errors.
These are single-window results conditioned on constructed assistant_speaking state. Published Jev scores now use the complete native Transformers run above; historical ms-swift results remain in the code repository under results/reference/.
Training and evaluation
The model was fine-tuned with LoRA rank 32, alpha 64, learning rate 5e-5, and a three-epoch budget. The processed pool contains 156,925 samples from Easy-Turn, Smart-Turn and MUSAN noise, with 141,232 assigned to training. On the separate internal test set of 3,923 samples, accuracy was 97.04% and seven-label Macro-F1 was 97.87%.
See EVALUATION.md for data distributions, label recalls, experiment provenance and timing boundaries.
Limitations
- Public sets do not assess background recall. Smart-Test's speaking group contains incomplete speech only, so its 99.90% accuracy does not establish interruption/backchannel discrimination.
- Performance is conditional on correct external playback state. These results do not establish streaming event latency or robustness to deployment devices, echo, noise and accents.
- This release requires a Qwen3-Omni-30B-A3B-Instruct GPU environment. CPU, ONNX and quantized deployments have not been benchmarked.
License and acknowledgements
Fine-tuned weights are released under Apache License 2.0, permitting no-charge research, personal and commercial use, modification and redistribution subject to its terms.
The base model was developed and released by the Qwen team: Qwen3-Omni-30B-A3B-Instruct. Its official model page declares Apache-2.0; the official license text is available in Qwen3-Omni/LICENSE. Base-model attribution and fine-tuning modifications are recorded in NOTICE.
Thanks to Qwen, ms-swift, OmniJev/OneJev, Easy-Turn, Smart-Turn and MUSAN. Datasets and dependencies retain their own licenses. This independent derivative is not endorsed by the upstream authors.
Source code and feedback
- Downloads last month
- 27
Model tree for Beagledata/Turn-Detection-Jev
Base model
Qwen/Qwen3-Omni-30B-A3B-Instruct
