---
license: bsd-2-clause
language: [en, hi, mr, bn, ta, te, kn, ml, gu, pa, or, as]
pipeline_tag: audio-classification
tags: [end-of-turn, turn-detection, voice-agent, pipecat, smart-turn, onnx, indic]
base_model: openai/whisper-base
datasets: [adimyth/indic-smart-turn-data, ai4bharat/IndicVoices, santhosh-005/tamil-eot, pipecat-ai/smart-turn-data-v3.2-train]
---
# Indic Smart Turn
End-of-turn detection for English and eleven Indian languages: Hindi, Marathi, Bengali, Tamil, Telugu, Kannada, Malayalam, Gujarati, Punjabi, Odia and Assamese. A voice agent has to decide at every pause whether the user has finished speaking. Indic Smart Turn makes that decision from the audio alone and runs on one CPU core in under 50 ms.
It is built with the [Pipecat Smart Turn v3](https://github.com/pipecat-ai/smart-turn) recipe from Whisper weights and trained on about 52,000 clips from real Indian phone conversations plus Pipecat's own data. It loads in Pipecat's `LocalSmartTurnAnalyzerV3` with a one-line change.
- **Models:** [adimyth/indic-smart-turn](https://huggingface.co/adimyth/indic-smart-turn)
- **Data:** [adimyth/indic-smart-turn-data](https://huggingface.co/datasets/adimyth/indic-smart-turn-data)
- **Release post:** [Building Indic Smart Turn](https://adimyth.in/essays/building-indic-smart-turn)
## Results
Accuracy at the 0.5 threshold on the same speaker-disjoint test clips, with ROC-AUC at the end of each row. Smart Turn v3.2 ships as a whisper-tiny in two files, int8 for CPU and fp32 for GPU, so each group compares with the v3.2 file of the same precision.
### Tiny int8 vs Smart Turn v3.2 int8
Same encoder size, precision and speed as the shipped CPU file.
### Base int8 vs Smart Turn v3.2 int8
### Tiny fp32 vs Smart Turn v3.2 fp32
### Base fp32 vs Smart Turn v3.2 fp32
The most accurate model. On the human-validated TamilEOT test it scores 85.9%, matching the paper's whisper-base result of 86.1%.
int8 results differ slightly between CPU architectures while fp32 results do not; every table here pairs models scored on one machine. Details: `int8_cpu_dependence.md` in this repo.
## Sales-training calls
12,710 pauses from 1,200 real sales-training calls, in which a trainee practises a pitch with a voice agent, across ten languages. The recordings are not published; only these numbers leave the machine. Accuracy / ROC-AUC at the 0.5 threshold:
| Language | Test clips | Smart Turn v3.2 int8 | Indic tiny int8 | Indic base int8 |
|---|---:|---|---|---|
| Tamil | 991 | 61.9 / 0.755 | 70.5 / 0.819 | 78.7 / 0.875 |
| Malayalam | 462 | 64.1 / 0.732 | 75.8 / 0.819 | 78.4 / 0.862 |
| Bengali | 2116 | 64.7 / 0.769 | 70.3 / 0.814 | 77.5 / 0.859 |
| Telugu | 1097 | 66.2 / 0.788 | 71.8 / 0.825 | 77.3 / 0.886 |
| Marathi | 925 | 66.4 / 0.780 | 73.4 / 0.820 | 77.0 / 0.854 |
| Odia * | 95 | 70.5 / 0.810 | 66.3 / 0.860 | 78.9 / 0.912 |
| Hindi | 2165 | 68.9 / 0.797 | 75.0 / 0.837 | 78.8 / 0.876 |
| Kannada | 367 | 64.3 / 0.778 | 67.8 / 0.819 | 74.1 / 0.864 |
| Gujarati | 2006 | 68.6 / 0.770 | 76.2 / 0.840 | 76.5 / 0.857 |
| English | 2486 | 67.6 / 0.752 | 71.2 / 0.794 | 75.2 / 0.829 |
| **All** | 12710 | **66.6 / 0.771** | **72.7 / 0.821** | **77.1 / 0.859** |
\* Odia: 95 clips from 2 sessions, indicative only.
The shipped base files were then tuned for one pass on 20,649 clips from these calls mixed with public data, with the decision threshold baked into the graph. Int8, accuracy / ROC-AUC, both rows scored on one machine:
| Test set | Version 1 | Tuned (shipped) |
|---|---|---|
| Sales-training calls, 12,710 clips | 77.1 / 0.859 | **79.4 / 0.875** |
| Public test, 20,431 clips | 89.9 / 0.955 | 89.8 / 0.958 |
The original files remain on Hugging Face with a `-v1` suffix.
## How to use it
Four files on Hugging Face:
- `indic-smart-turn-base-int8.onnx`, recommended on x86 servers
- `indic-smart-turn-base-fp32.onnx`, for Apple Silicon and GPUs
- `indic-smart-turn-tiny-int8.onnx`
- `indic-smart-turn-tiny-fp32.onnx`
All take the same input as Smart Turn v3, 8 s of 16 kHz audio as an 80 x 800 log-mel, and return the probability that the turn is complete.
### Pipecat
```python
from pipecat.audio.turn.smart_turn.local_smart_turn_v3 import LocalSmartTurnAnalyzerV3
analyzer = LocalSmartTurnAnalyzerV3(smart_turn_model_path="indic-smart-turn-base-int8.onnx")
```
### Directly
The model takes an 80 x 800 log-mel of the last 8 seconds of 16 kHz mono audio and returns the probability that the turn is complete. `config.json` records the same contract.
Single clip:
```python
import numpy as np, onnxruntime as ort, soundfile as sf
from transformers import WhisperFeatureExtractor
MODEL = "indic-smart-turn-base-int8.onnx"
fe = WhisperFeatureExtractor(chunk_length=8) # 8 s window, 80 mel bins, 800 frames
so = ort.SessionOptions(); so.intra_op_num_threads = 1 # one core is enough at batch 1
session = ort.InferenceSession(MODEL, so, providers=["CPUExecutionProvider"])
def last_8s(audio: np.ndarray, sr: int = 16000) -> np.ndarray:
n = 8 * sr
return audio[-n:] if len(audio) > n else np.pad(audio, (n - len(audio), 0)) # keep the end, zero-pad the front
def turn_complete_prob(audio: np.ndarray) -> float:
feats = fe(last_8s(audio), sampling_rate=16000, return_tensors="np", padding="max_length",
max_length=8 * 16000, truncation=True, do_normalize=True).input_features.astype(np.float32)
return float(session.run(None, {"input_features": feats})[0][0, 0])
audio, sr = sf.read("clip.wav", dtype="float32") # 16 kHz mono; resample first if not
p = turn_complete_prob(audio)
print("complete" if p > 0.5 else "incomplete", round(p, 3))
```
## License and attribution
- **Code and models: BSD 2-Clause**, the same license as Pipecat Smart Turn. Code: [github.com/adimyth/indic-smart-turn](https://github.com/adimyth/indic-smart-turn).
- **Dataset `adimyth/indic-smart-turn-data`: CC BY 4.0**, because its audio comes from IndicVoices, which is CC BY 4.0. Credit AI4Bharat when you use it.
- [Pipecat Smart Turn v3](https://github.com/pipecat-ai/smart-turn) by Pipecat / Daily: architecture, trainer and the v3.2 training data (CC BY 4.0).
- [ai4bharat/IndicVoices](https://huggingface.co/datasets/ai4bharat/IndicVoices) by AI4Bharat: conversational audio (CC BY 4.0).
- [santhosh-005/tamil-eot](https://huggingface.co/datasets/santhosh-005/tamil-eot): Tamil turn boundaries and the recipe this project follows (CC BY 4.0).
