File size: 7,834 Bytes
81a0b99 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 | ---
license: bsd-2-clause
language:
- ur
library_name: onnx
pipeline_tag: audio-classification
base_model: adimyth/indic-smart-turn
datasets:
- ai4bharat/IndicVoices
- adimyth/indic-smart-turn-data
tags:
- turn-detection
- end-of-turn
- voice-agents
- pipecat
- smart-turn
- urdu
- onnx
- whisper
---
<p align="center">
<img alt="baari" src="https://raw.githubusercontent.com/samadpls/baari/main/assets/hero.png" width="100%">
</p>
<p align="center">
<a href="https://github.com/samadpls/baari"><img alt="Code on GitHub" src="https://img.shields.io/badge/code-samadpls%2Fbaari-0E6B4F?logo=github"></a>
<a href="https://pypi.org/project/baari/"><img alt="PyPI" src="https://img.shields.io/pypi/v/baari?color=0E6B4F&label=pypi"></a>
<a href="https://samadpls.com"><img alt="by samadpls.com" src="https://img.shields.io/badge/by-samadpls.com-0E6B4F"></a>
<img alt="License: BSD-2-Clause" src="https://img.shields.io/badge/license-BSD--2--Clause-F2B544">
</p>
# baari
**End-of-turn detection for Urdu voice agents.** At every pause, baari decides whether the speaker has
finished (reply now) or is only pausing mid-sentence (keep listening). baari (باری) means "turn" in Urdu.
It is Indic Smart Turn tiny, fine-tuned on Urdu phone conversations. It takes the same input and gives
the same output as [Smart Turn](https://github.com/pipecat-ai/smart-turn), so in Pipecat it replaces
Smart Turn with one line.
## 📊 Results
The test set has 1,588 clips of real Urdu phone conversations from the held-out `valid` split of
IndicVoices: 1,470 where the speaker had finished and 118 where they were pausing mid-sentence. No test
speaker, and neither side of any test call, is in the training data. Because most clips are "finished",
the fair score is balanced accuracy, the average of the two right-hand columns. A model that always
says "finished" gets 50%.
| Model | Balanced accuracy (95% CI) | Waits when unfinished | Replies when finished |
|---|---|---|---|
| **baari int8** (default) | **77.0%** (72.9–81.0) | 62.7% | 91.4% |
| **baari fp32** | **76.8%** (72.4–80.7) | 63.6% | 89.9% |
| Indic Smart Turn tiny (int8) | 73.4% (69.0–77.8) | 61.0% | 85.9% |
| Indic Smart Turn base (fp32) | 70.9% (66.5–75.4) | 47.5% | 94.4% |
| Indic Smart Turn tiny (fp32) | 70.0% (65.8–74.4) | 51.7% | 88.4% |
| Smart Turn v3.2 (CPU, int8) | 63.4% (58.8–68.1) | 57.6% | 69.2% |
| Smart Turn v3.2 (GPU, fp32) | 58.8% (54.8–63.0) | 76.3% | 41.3% |
| PuristanLabs1/urdu-turn-v2 | 51.8% (48.4–55.4) | 16.9% | 86.7% |
- Scored on the CPU of a MacBook Pro (M3 Pro), with the same audio preparation Pipecat uses.
- **int8 scores depend on the CPU.** baari int8 gets 77.0% on Arm (Apple) and 74.9% on x86 (Intel and
AMD, which most servers use). fp32 gets 76.8% on both.
- On the same clips, baari fp32 beats Indic Smart Turn tiny int8 by 3.3 points (paired bootstrap, 95% CI
+0.3 to +6.4), and its own starting point, Indic Smart Turn tiny fp32, by 6.7 points (+3.8 to +10.0).
## 🚀 Use it
```bash
pip install baari
```
```python
from baari import TurnDetector
detector = TurnDetector(threads=4) # downloads baari-int8.onnx once
detector.is_finished(audio, sample_rate=16000) # mono audio (float or int16) that ends at the pause
detector.probability(audio, sample_rate=16000) # the same decision as a number from 0 to 1
```
In [Pipecat](https://github.com/pipecat-ai/pipecat):
```python
from baari import model_path
from pipecat.audio.turn.smart_turn.local_smart_turn_v3 import LocalSmartTurnAnalyzerV3
turn_analyzer = LocalSmartTurnAnalyzerV3(smart_turn_model_path=model_path())
```
One call, including the audio preparation, on the CPU of a MacBook Pro (M3 Pro):
| File | 1 thread | 4 threads |
|---|---|---|
| int8 (default) | 36 ms | 12 ms |
| fp32: `TurnDetector(precision="fp32")` | 77 ms | 23 ms |
Use 1 thread on a server that handles many calls at once. baari needs 16 kHz audio, so resample 8 kHz
phone audio first, for example with `soxr.resample(audio, 8000, 16000)`.
<details>
<summary>Files, input and output</summary>
| File | What it is |
|---|---|
| `baari-int8.onnx` | 8.8 MB, the default |
| `baari-fp32.onnx` | 32 MB, full precision |
| `baari.pt` | PyTorch weights, in the layout of `SmartTurn` in the code's `train.py` |
| `metrics.json` | The training record: data sizes, settings, and dev and test scores after each epoch |
- **Input** `input_features`, float32 `[batch, 80, 800]`: the Whisper log-mel spectrogram of the last
8 seconds of 16 kHz audio, zero-padded at the front, with zero-mean, unit-variance normalisation.
- **Output** `logits`, float32 `[batch, 1]`: despite the name, the probability that the speaker has
finished, as in Smart Turn v3. Above 0.5 means "reply now".
</details>
<details>
<summary>How it was trained</summary>
- **Start:** Indic Smart Turn tiny fp32, a Whisper-tiny encoder with attention pooling and a small
classifier (8M parameters). It ships only as ONNX, so its weights were read back into PyTorch. A fresh
model with those weights matches the ONNX file to within 6e-6.
- **Urdu data:** 57,182 clips from the conversations in the IndicVoices `train` split. Test speakers and
both sides of every test call are left out.
- **Mixed in:** 39,925 clips from the training split of Indic Smart Turn's data, which covers 11 other
Indian languages, so the model keeps what it already knew. 17,417 of the 97,107 training clips are "unfinished".
- **Labels:** an LLM agent read each call's transcript in order and marked whether the speaker had
finished at the end of each segment, using
[this prompt](https://github.com/samadpls/baari/blob/main/prompts/label.md).
- **Clips:** the last 8 seconds of audio ending at the segment, with the speaker's earlier segments as
context. This is Indic Smart Turn's recipe.
- **Training:** 3 epochs, AdamW, learning rate 2e-5 with warm-up and linear decay, batch 64, fp16, on
one T4 GPU in 34 minutes. Each batch weights the classes by how many of each it has, as in Smart Turn.
- **Choosing the epoch:** the best epoch (the 2nd) was picked on an Urdu dev set of 4,069 clips from
speakers held out of training. The test set played no part in any choice.
- **int8:** static quantization with Smart Turn's settings: QDQ, per-channel int8 weights, uint8
activations, calibrated on 1,024 training clips.
</details>
## ⚠️ Limits
- **The labels come from transcripts, not audio.** A native speaker listened to 50 test clips. All 25
"finished" labels were right, and 19 of 25 "unfinished" labels were. So about 1 in 4 "unfinished"
labels may be wrong, in the training data and in the test set, and "waits when unfinished" is a rough
measure.
- **Tested only on IndicVoices phone conversations.** baari has not been measured on other recording
conditions, or on other kinds of Urdu speech.
- **In the training clips, the gaps and the last 0.2 seconds are digital silence.** Real calls have
background noise there.
- **int8 results vary by about 2 points between CPU types.** Use fp32 if you need the same answer on
every machine.
## Credits and license
The model is BSD-2-Clause, like the models it builds on:
[Indic Smart Turn](https://github.com/adimyth/indic-smart-turn) (BSD-2-Clause),
[Smart Turn](https://github.com/pipecat-ai/smart-turn) by Pipecat (BSD-2-Clause) and
[Whisper](https://huggingface.co/openai/whisper-tiny) by OpenAI (Apache-2.0). It was trained on
[IndicVoices](https://huggingface.co/datasets/ai4bharat/IndicVoices) by AI4Bharat and
[Indic Smart Turn's data](https://huggingface.co/datasets/adimyth/indic-smart-turn-data), both CC BY 4.0.
The code to rebuild the data and the model is at [github.com/samadpls/baari](https://github.com/samadpls/baari).
Made by [Abdul Samad](https://samadpls.com) (abdulsamadsid1@gmail.com).
|