README.md

---
language: [en, hi]
license: bsd-2-clause
tags:
  - turn-detection
  - endpointing
  - voice-activity-detection
  - audio-classification
  - hinglish
  - onnx
pipeline_tag: audio-classification
---

# Smart Turn Hinglish (whisper-tiny)

Audio-only turn detection for voice agents: has the speaker **finished their turn**, or are
they just pausing? Built for English + Hindi/Hinglish from the
[pipecat smart-turn-v3.2](https://huggingface.co/datasets/pipecat-ai/smart-turn-data-v3.2-train)
data. ~7.8M params. The ONNX graph takes a raw 16k waveform and returns `P(turn complete)`,
with the log-mel frontend baked in, so inference needs only `onnxruntime` + `numpy`.

**Code, training, ablations and the full write-up:**
[github.com/CodeWithMoin/smart-turn-hinglish](https://github.com/CodeWithMoin/smart-turn-hinglish)

## Files

| file | window | notes |
|---|---|---|
| `smart_turn_8s.onnx` | 8s | shipped model. accuracy pick, ~38 ms/clip CPU |
| `smart_turn_4s.onnx` | 4s | fast variant, ~17 ms/clip CPU |

## Results (official frozen `smart-turn-v3.2-test`, accuracy @ 0.5)

| | this model (8s) | Smart Turn v3.2 |
|---|---|---|
| Hindi | **93.9%** | 92.8% |
| English | 93.7% | 94.7% |
| Human / real | 94.4% | 95.5% |

Beats the reference on Hindi and sits within a point on English, at a fraction of the size.
Overall test AUC 0.983.

## Usage

```python
import numpy as np, onnxruntime as ort

sess = ort.InferenceSession("smart_turn_8s.onnx", providers=["CPUExecutionProvider"])

def p_turn_complete(wav_16k):                 # float32 mono @ 16 kHz
    x = wav_16k[-128000:]                     # last 8s
    buf = np.zeros((1, 128000), np.float32)
    buf[0, 128000 - len(x):] = x              # right-align (left-pad with silence)
    return float(sess.run(None, {"waveform": buf})[0][0])
```

In a live agent, run a cheap VAD (e.g. Silero) continuously; on a ~200 ms pause, call this on
the last 8s. Threshold `P(complete)` — higher = more patient (fewer interruptions).

## Limitations

Trained mostly on synthetic TTS. Real casual conversational speech on a laptop/phone mic is
out of distribution and where it's weakest. See the report for the honest failure analysis
and the plan (real Indian-mic conversational data) to close it.

## Provenance

Fine-tuned from `openai/whisper-tiny` on the openly published pipecat smart-turn-v3.2 corpus.
For commercial use, check the upstream terms.