| --- |
| language: [en, hi] |
| license: bsd-2-clause |
| tags: |
| - turn-detection |
| - endpointing |
| - voice-activity-detection |
| - audio-classification |
| - hinglish |
| - onnx |
| pipeline_tag: audio-classification |
| --- |
| |
| # Smart Turn Hinglish (whisper-tiny) |
|
|
| Audio-only turn detection for voice agents: has the speaker **finished their turn**, or are |
| they just pausing? Built for English + Hindi/Hinglish from the |
| [pipecat smart-turn-v3.2](https://huggingface.co/datasets/pipecat-ai/smart-turn-data-v3.2-train) |
| data. ~7.8M params. The ONNX graph takes a raw 16k waveform and returns `P(turn complete)`, |
| with the log-mel frontend baked in, so inference needs only `onnxruntime` + `numpy`. |
|
|
| **Code, training, ablations and the full write-up:** |
| [github.com/CodeWithMoin/smart-turn-hinglish](https://github.com/CodeWithMoin/smart-turn-hinglish) |
|
|
| ## Files |
|
|
| | file | window | notes | |
| |---|---|---| |
| | `smart_turn_8s.onnx` | 8s | shipped model. accuracy pick, ~38 ms/clip CPU | |
| | `smart_turn_4s.onnx` | 4s | fast variant, ~17 ms/clip CPU | |
|
|
| ## Results (official frozen `smart-turn-v3.2-test`, accuracy @ 0.5) |
|
|
| | | this model (8s) | Smart Turn v3.2 | |
| |---|---|---| |
| | Hindi | **93.9%** | 92.8% | |
| | English | 93.7% | 94.7% | |
| | Human / real | 94.4% | 95.5% | |
|
|
| Beats the reference on Hindi and sits within a point on English, at a fraction of the size. |
| Overall test AUC 0.983. |
|
|
| ## Usage |
|
|
| ```python |
| import numpy as np, onnxruntime as ort |
| |
| sess = ort.InferenceSession("smart_turn_8s.onnx", providers=["CPUExecutionProvider"]) |
| |
| def p_turn_complete(wav_16k): # float32 mono @ 16 kHz |
| x = wav_16k[-128000:] # last 8s |
| buf = np.zeros((1, 128000), np.float32) |
| buf[0, 128000 - len(x):] = x # right-align (left-pad with silence) |
| return float(sess.run(None, {"waveform": buf})[0][0]) |
| ``` |
|
|
| In a live agent, run a cheap VAD (e.g. Silero) continuously; on a ~200 ms pause, call this on |
| the last 8s. Threshold `P(complete)` — higher = more patient (fewer interruptions). |
|
|
| ## Limitations |
|
|
| Trained mostly on synthetic TTS. Real casual conversational speech on a laptop/phone mic is |
| out of distribution and where it's weakest. See the report for the honest failure analysis |
| and the plan (real Indian-mic conversational data) to close it. |
|
|
| ## Provenance |
|
|
| Fine-tuned from `openai/whisper-tiny` on the openly published pipecat smart-turn-v3.2 corpus. |
| For commercial use, check the upstream terms. |
|
|
