Release Indic Smart Turn

.gitattributes

@@ -0,0 +1,36 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+pause-test.mp4 filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,103 @@
+---
+license: bsd-2-clause
+language: [en, hi, mr, bn, ta, te, kn, ml, gu, pa, or, as]
+pipeline_tag: audio-classification
+tags: [end-of-turn, turn-detection, voice-agent, pipecat, smart-turn, onnx, indic]
+base_model: openai/whisper-base
+datasets: [adimyth/indic-smart-turn-data, ai4bharat/IndicVoices, santhosh-005/tamil-eot, pipecat-ai/smart-turn-data-v3.2-train]
+---
+
+# Indic Smart Turn
+
+End-of-turn detection for English and eleven Indian languages: Hindi, Marathi, Bengali, Tamil, Telugu, Kannada, Malayalam, Gujarati, Punjabi, Odia and Assamese. A voice agent has to decide at every pause whether the user has finished speaking. Indic Smart Turn makes that decision from the audio alone and runs on one CPU core in under 50 ms.
+
+It is built with the [Pipecat Smart Turn v3](https://github.com/pipecat-ai/smart-turn) recipe from Whisper weights and trained mostly on real Indian phone conversations from IndicVoices and TamilEOT, plus Pipecat's own data. It loads in Pipecat's `LocalSmartTurnAnalyzerV3` with a one-line change.
+
+- **Models:** [adimyth/indic-smart-turn](https://huggingface.co/adimyth/indic-smart-turn)
+- **Data:** [adimyth/indic-smart-turn-data](https://huggingface.co/datasets/adimyth/indic-smart-turn-data)
+- **Release post:** [Building Indic Smart Turn](https://adimyth.in/essays/building-indic-smart-turn)
+
+<video src="https://huggingface.co/adimyth/indic-smart-turn/resolve/main/pause-test.mp4" controls muted width="100%"></video>
+
+## Results
+
+Accuracy at the 0.5 threshold on the same speaker-disjoint test clips, with ROC-AUC at the end of each row. Smart Turn v3.2 ships as a whisper-tiny in two files, int8 for CPU and fp32 for GPU, so each group compares with the v3.2 file of the same precision.
+
+### Tiny int8 vs Smart Turn v3.2 int8
+
+Same encoder size, precision and speed as the shipped CPU file.
+
+![tiny int8 vs Smart Turn v3.2 int8](https://huggingface.co/adimyth/indic-smart-turn/resolve/main/figures/group1_tiny_int8-light.svg)
+
+### Base int8 vs Smart Turn v3.2 int8
+
+![base int8 vs Smart Turn v3.2 int8](https://huggingface.co/adimyth/indic-smart-turn/resolve/main/figures/group2_base_int8-light.svg)
+
+### Tiny fp32 vs Smart Turn v3.2 fp32
+
+![tiny fp32 vs Smart Turn v3.2 fp32](https://huggingface.co/adimyth/indic-smart-turn/resolve/main/figures/group3_tiny_fp32-light.svg)
+
+### Base fp32 vs Smart Turn v3.2 fp32
+
+On the human-validated TamilEOT test it scores 85.9%, close to the paper's whisper-base result of 86.1%.
+
+![base fp32 vs Smart Turn v3.2 fp32](https://huggingface.co/adimyth/indic-smart-turn/resolve/main/figures/group4_base_fp32-light.svg)
+
+int8 results differ slightly between CPU architectures while fp32 results do not, so each int8 chart pairs models scored on one machine. Details: `int8_cpu_dependence.md` in this repo.
+
+## How to use it
+
+Four files on Hugging Face:
+
+- `indic-smart-turn-base-int8.onnx`, recommended on x86 servers
+- `indic-smart-turn-base-fp32.onnx`, for Apple Silicon and GPUs
+- `indic-smart-turn-tiny-int8.onnx`
+- `indic-smart-turn-tiny-fp32.onnx`
+
+All take the same input as Smart Turn v3, 8 s of 16 kHz audio as an 80 x 800 log-mel, and return the probability that the turn is complete.
+
+### Pipecat
+
+```python
+from pipecat.audio.turn.smart_turn.local_smart_turn_v3 import LocalSmartTurnAnalyzerV3
+
+analyzer = LocalSmartTurnAnalyzerV3(smart_turn_model_path="indic-smart-turn-base-int8.onnx")
+```
+
+
+### Directly
+
+The model takes an 80 x 800 log-mel of the last 8 seconds of 16 kHz mono audio and returns the probability that the turn is complete. `config.json` records the same contract.
+
+Single clip:
+
+```python
+import numpy as np, onnxruntime as ort, soundfile as sf
+from transformers import WhisperFeatureExtractor
+
+MODEL = "indic-smart-turn-base-int8.onnx"
+fe = WhisperFeatureExtractor(chunk_length=8)           # 8 s window, 80 mel bins, 800 frames
+so = ort.SessionOptions(); so.intra_op_num_threads = 1  # one core is enough at batch 1
+session = ort.InferenceSession(MODEL, so, providers=["CPUExecutionProvider"])
+
+def last_8s(audio: np.ndarray, sr: int = 16000) -> np.ndarray:
+    n = 8 * sr
+    return audio[-n:] if len(audio) > n else np.pad(audio, (n - len(audio), 0))  # keep the end, zero-pad the front
+
+def turn_complete_prob(audio: np.ndarray) -> float:
+    feats = fe(last_8s(audio), sampling_rate=16000, return_tensors="np", padding="max_length",
+               max_length=8 * 16000, truncation=True, do_normalize=True).input_features.astype(np.float32)
+    return float(session.run(None, {"input_features": feats})[0][0, 0])
+
+audio, sr = sf.read("clip.wav", dtype="float32")        # 16 kHz mono; resample first if not
+p = turn_complete_prob(audio)
+print("complete" if p > 0.5 else "incomplete", round(p, 3))
+```
+
+## License and attribution
+
+- **Code and models: BSD 2-Clause**, the same license as Pipecat Smart Turn. Code: [github.com/adimyth/indic-smart-turn](https://github.com/adimyth/indic-smart-turn).
+- **Dataset `adimyth/indic-smart-turn-data`: CC BY 4.0**, because its audio comes from IndicVoices, which is CC BY 4.0. Credit AI4Bharat when you use it.
+- [Pipecat Smart Turn v3](https://github.com/pipecat-ai/smart-turn) by Pipecat / Daily: architecture, trainer and the v3.2 training data (CC BY 4.0).
+- [ai4bharat/IndicVoices](https://huggingface.co/datasets/ai4bharat/IndicVoices) by AI4Bharat: conversational audio (CC BY 4.0).
+- [santhosh-005/tamil-eot](https://huggingface.co/datasets/santhosh-005/tamil-eot): Tamil turn boundaries and the recipe this project follows (CC BY 4.0).

charts.html

@@ -0,0 +1,89 @@
+<title>Indic Smart Turn Results</title>
+<link rel="stylesheet" href="https://fonts.googleapis.com/css2?family=IBM+Plex+Sans:wght@400;500;600&family=IBM+Plex+Mono:wght@400;500&display=swap">
+<style>
+/* Layout: one column of four small-multiple bar charts, each self-contained with legend, hover tooltip and a table view. */
+:root{
+  --bg:#fcfcfb; --surface:#f3f2ee; --ink:#141412; --ink-2:#55534d; --ink-3:#8a877f; --rule:#dedcd5; --grid:#e9e8e3;
+  --ours:#2a78d6; --base:#eb6834; --ours-ink:#1c5cab; --base-ink:#b8461f;
+  --font-body:"IBM Plex Sans",system-ui,-apple-system,"Segoe UI",sans-serif; --font-mono:"IBM Plex Mono",ui-monospace,Menlo,monospace;
+}
+@media (prefers-color-scheme: dark){ :root:not([data-theme="light"]){ color-scheme:dark; --bg:#1a1a19; --surface:#232322; --ink:#f3f2ee; --ink-2:#c3c2b7; --ink-3:#8f8d85; --rule:#3a3936; --grid:#2d2d2b; --ours:#3987e5; --base:#d95926; --ours-ink:#86b6ef; --base-ink:#f2a07f; } }
+:root[data-theme="dark"]{ color-scheme:dark; --bg:#1a1a19; --surface:#232322; --ink:#f3f2ee; --ink-2:#c3c2b7; --ink-3:#8f8d85; --rule:#3a3936; --grid:#2d2d2b; --ours:#3987e5; --base:#d95926; --ours-ink:#86b6ef; --base-ink:#f2a07f; }
+body{margin:0;background:var(--bg);color:var(--ink);font-family:var(--font-body);font-size:15px;line-height:1.5}
+.wrap{max-width:980px;margin:0 auto;padding-block:32px 56px;padding-inline:20px}
+h1{font-size:26px;font-weight:600;margin:0 0 6px;text-wrap:balance;letter-spacing:-0.01em}
+.lede{color:var(--ink-2);max-width:66ch;margin:0 0 8px}
+.meta{font-family:var(--font-mono);font-size:12px;color:var(--ink-3);letter-spacing:.02em;text-transform:uppercase;margin-bottom:28px}
+.chart{border-top:1px solid var(--rule);padding-block:22px 10px}
+.chart h2{font-size:17px;font-weight:600;margin:0 0 4px;text-wrap:balance}
+.chart .sub{color:var(--ink-2);margin:0 0 12px;max-width:70ch}
+.legend{display:flex;flex-wrap:wrap;gap:18px;font-size:13px;color:var(--ink-2);margin:0 0 10px}
+.legend span{display:inline-flex;align-items:center;gap:8px}
+.legend i{display:inline-block;width:14px;height:10px;border-radius:2px}
+.plot{position:relative}
+svg{display:block;width:100%;height:auto}
+.bar{cursor:default}
+.bar:hover{filter:brightness(1.12)}
+.tip{position:absolute;pointer-events:none;background:var(--ink);color:var(--bg);font-family:var(--font-mono);font-size:12px;padding:6px 9px;border-radius:4px;white-space:nowrap;transform:translate(-50%,-115%);opacity:0;transition:opacity .08s}
+.tip.on{opacity:1}
+.tools{display:flex;gap:10px;align-items:center;margin-top:6px}
+button{font:inherit;font-size:13px;color:var(--ink-2);background:var(--surface);border:1px solid var(--rule);border-radius:4px;padding:5px 10px;cursor:pointer}
+button:hover,button:focus-visible{color:var(--ink);outline:2px solid var(--ours);outline-offset:1px}
+table{border-collapse:collapse;width:100%;margin-top:10px;font-size:13px;font-variant-numeric:tabular-nums}
+th,td{text-align:right;padding:5px 8px;border-bottom:1px solid var(--grid)}
+th:first-child,td:first-child{text-align:left}
+th{color:var(--ink-3);font-weight:500}
+.pos{color:var(--ours-ink)} .neg{color:var(--base-ink)}
+.note{color:var(--ink-3);font-size:12.5px;max-width:70ch;margin-top:24px}
+@media (max-width:520px){ .wrap{padding-inline:16px} h1{font-size:22px} }
+@media (prefers-reduced-motion: reduce){ .tip{transition:none} }
+</style>
+<div class="wrap">
+  <h1>Indic Smart Turn versus Smart Turn v3.2</h1>
+  <p class="lede">Accuracy per language on the same speaker-disjoint test clips, at the default 0.5 threshold. Hover a bar for the exact value; open the table for the numbers and the difference.</p>
+  <div class="meta">11 Indian languages + English · 20,431 test clips · four model groups</div>
+  <div id="charts"></div>
+  <p class="note">Pipecat ships Smart Turn v3.2 as two whisper-tiny files: int8 for CPU and fp32 for GPU. There is no base-size v3.2, so the base groups are compared with the tiny v3.2 file of the same precision. English, Hindi, Marathi and Bengali rows include Pipecat's own v3.2 test set; Tamil includes the human-validated TamilEOT test set.</p>
+</div>
+<script>
+const DATA = {"group1_tiny_int8": {"title": "Group 1: tiny int8 (8.7 MB) vs Smart Turn v3.2 int8 (same size, same precision)", "ours_label": "Indic Smart Turn tiny int8", "baseline_label": "Smart Turn v3.2 int8 (CPU file, whisper-tiny)", "languages": ["English", "Hindi", "Marathi", "Bengali", "Tamil", "Telugu", "Kannada", "Malayalam", "Gujarati", "Punjabi", "Odia", "Assamese"], "ours": [91.7, 89.4, 84.1, 84.3, 82.1, 85.1, 82.9, 80.9, 80.9, 80.2, 86.4, 80.4], "baseline": [92.9, 84.7, 77.9, 81.2, 70.5, 73.4, 77.7, 66.9, 73.5, 69.2, 74.6, 67.8]}, "group2_base_int8": {"title": "Group 2: base int8 (24 MB) vs Smart Turn v3.2 int8 (same precision; v3.2 has no base size)", "ours_label": "Indic Smart Turn base int8", "baseline_label": "Smart Turn v3.2 int8 (CPU file, whisper-tiny)", "languages": ["English", "Hindi", "Marathi", "Bengali", "Tamil", "Telugu", "Kannada", "Malayalam", "Gujarati", "Punjabi", "Odia", "Assamese"], "ours": [93.9, 91.5, 88.1, 86.8, 86.7, 84.8, 89.1, 85.6, 87.1, 86.5, 87.3, 83.3], "baseline": [94.3, 85.8, 77.7, 81.4, 70.0, 70.9, 74.0, 63.8, 72.5, 69.4, 75.2, 64.9], "ours_auc": [0.988, 0.971, 0.955, 0.936, 0.922, 0.923, 0.92, 0.873, 0.896, 0.892, 0.933, 0.892], "baseline_auc": [0.983, 0.941, 0.865, 0.884, 0.749, 0.766, 0.783, 0.664, 0.765, 0.749, 0.778, 0.691]}, "group3_tiny_fp32": {"title": "Group 3: tiny fp32 (32 MB) vs Smart Turn v3.2 fp32 (same size, same precision)", "ours_label": "Indic Smart Turn tiny fp32", "baseline_label": "Smart Turn v3.2 fp32 (GPU file, whisper-tiny)", "languages": ["English", "Hindi", "Marathi", "Bengali", "Tamil", "Telugu", "Kannada", "Malayalam", "Gujarati", "Punjabi", "Odia", "Assamese"], "ours": [93.7, 90.2, 88.1, 85.4, 83.4, 85.0, 82.3, 81.4, 82.5, 83.1, 84.9, 81.4], "baseline": [94.7, 83.0, 75.8, 75.9, 58.6, 58.1, 56.6, 49.6, 58.4, 56.7, 60.4, 55.4]}, "group4_base_fp32": {"title": "Group 4: base fp32 (81 MB) vs Smart Turn v3.2 fp32 (same precision; v3.2 has no base size)", "ours_label": "Indic Smart Turn base fp32", "baseline_label": "Smart Turn v3.2 fp32 (GPU file, whisper-tiny)", "languages": ["English", "Hindi", "Marathi", "Bengali", "Tamil", "Telugu", "Kannada", "Malayalam", "Gujarati", "Punjabi", "Odia", "Assamese"], "ours": [94.0, 91.4, 88.0, 86.5, 86.4, 85.1, 89.6, 85.4, 86.9, 86.9, 87.3, 83.3], "baseline": [94.7, 83.0, 75.8, 75.9, 58.6, 58.1, 56.6, 49.6, 58.4, 56.7, 60.4, 55.4], "ours_auc": [0.988, 0.97, 0.954, 0.938, 0.921, 0.922, 0.92, 0.876, 0.898, 0.896, 0.935, 0.891], "baseline_auc": [0.986, 0.943, 0.88, 0.893, 0.744, 0.773, 0.795, 0.667, 0.759, 0.748, 0.785, 0.678]}};
+const order = ["group1_tiny_int8","group2_base_int8","group3_tiny_fp32","group4_base_fp32"];
+const subs = {
+  group1_tiny_int8: "Same encoder size and precision as the shipped Smart Turn v3.2 CPU file.",
+  group2_base_int8: "Weights-only int8; ROC-AUC within 0.005 of fp32 in every language.",
+  group3_tiny_fp32: "Compared with the Smart Turn v3.2 GPU file, which is fp32.",
+  group4_base_fp32: "On TamilEOT it scores 85.9%, close to the paper's whisper-base result of 86.1%."
+};
+const root = document.getElementById("charts");
+for (const key of order) {
+  const d = DATA[key]; const n = d.languages.length;
+  const W = 900, L = 96, R = 60, T = 8, rowH = 34, H = T + n*rowH + 30, x0 = 50, x1 = 100;
+  const sx = v => L + (v - x0) / (x1 - x0) * (W - L - R);
+  let svg = `<svg viewBox="0 0 ${W} ${H}" role="img" aria-label="${d.title}">`;
+  for (let g = x0; g <= x1; g += 10) svg += `<line x1="${sx(g)}" x2="${sx(g)}" y1="${T}" y2="${H-26}" stroke="var(--grid)" stroke-width="1"/><text x="${sx(g)}" y="${H-10}" text-anchor="middle" font-size="12" fill="var(--ink-3)" font-family="var(--font-mono)">${g}</text>`;
+  d.languages.forEach((lang, i) => {
+    const y = T + i*rowH, o = d.ours[i], b = d.baseline[i], delta = (o - b);
+    svg += `<text x="${L-10}" y="${y+rowH/2+4}" text-anchor="end" font-size="13" fill="var(--ink)">${lang}</text>`;
+    svg += `<rect class="bar" data-tip="${d.ours_label}: ${o.toFixed(1)}%" x="${sx(x0)}" y="${y+4}" width="${sx(o)-sx(x0)}" height="11" rx="2" fill="var(--ours)"/>`;
+    svg += `<rect class="bar" data-tip="${d.baseline_label}: ${b.toFixed(1)}%" x="${sx(x0)}" y="${y+18}" width="${sx(b)-sx(x0)}" height="11" rx="2" fill="var(--base)"/>`;
+    svg += `<text x="${sx(Math.max(o,b))+8}" y="${y+rowH/2+4}" font-size="12" fill="var(--ink-2)" font-family="var(--font-mono)">${delta>=0?"+":""}${delta.toFixed(1)}</text>`;
+  });
+  svg += `</svg>`;
+  let table = `<table hidden><thead><tr><th>Language</th><th>${d.ours_label}</th><th>${d.baseline_label}</th><th>Difference</th></tr></thead><tbody>`;
+  d.languages.forEach((lang,i)=>{ const dl=d.ours[i]-d.baseline[i]; table += `<tr><td>${lang}</td><td>${d.ours[i].toFixed(1)}</td><td>${d.baseline[i].toFixed(1)}</td><td class="${dl>=0?'pos':'neg'}">${dl>=0?'+':''}${dl.toFixed(1)}</td></tr>`; });
+  table += `</tbody></table>`;
+  const sec = document.createElement("section"); sec.className = "chart";
+  sec.innerHTML = `<h2>${d.title}</h2><p class="sub">${subs[key]}</p>
+    <div class="legend"><span><i style="background:var(--ours)"></i>${d.ours_label}</span><span><i style="background:var(--base)"></i>${d.baseline_label}</span><span><i style="background:transparent;width:0"></i>Number at the end of each row: difference in points</span></div>
+    <div class="plot">${svg}<div class="tip"></div></div>
+    <div class="tools"><button type="button" id="btn-${key}" aria-expanded="false">Show table</button></div>${table}`;
+  root.appendChild(sec);
+  const tip = sec.querySelector(".tip"), plot = sec.querySelector(".plot");
+  sec.querySelectorAll(".bar").forEach(bar => {
+    bar.addEventListener("mousemove", e => { const r = plot.getBoundingClientRect(); tip.textContent = bar.dataset.tip; tip.style.left = (e.clientX - r.left) + "px"; tip.style.top = (e.clientY - r.top) + "px"; tip.classList.add("on"); });
+    bar.addEventListener("mouseleave", () => tip.classList.remove("on"));
+  });
+  const btn = sec.querySelector("button"), tbl = sec.querySelector("table");
+  btn.addEventListener("click", () => { tbl.hidden = !tbl.hidden; btn.textContent = tbl.hidden ? "Show table" : "Hide table"; btn.setAttribute("aria-expanded", String(!tbl.hidden)); });
+}
+</script>

config.json

@@ -0,0 +1,13 @@
+{
+  "architecture": "whisper encoder + attention pooling + MLP head (Pipecat Smart Turn v3 layout)",
+  "input": {"name": "input_features", "shape": [-1, 80, 800], "description": "log-mel of the last 8 s of 16 kHz audio, WhisperFeatureExtractor(chunk_length=8), do_normalize=True"},
+  "output": {"name": "logits", "shape": [-1, 1], "description": "probability that the user's turn is complete"},
+  "threshold": 0.5,
+  "languages": ["eng", "hin", "mar", "ben", "tam", "tel", "kan", "mal", "guj", "pan", "ori", "asm"],
+  "files": {
+    "indic-smart-turn-base-int8.onnx": {"encoder": "whisper-base", "precision": "int8 dynamic (weights)", "size_mb": 24, "recommended": true},
+    "indic-smart-turn-base-fp32.onnx": {"encoder": "whisper-base", "precision": "fp32", "size_mb": 81},
+    "indic-smart-turn-tiny-int8.onnx": {"encoder": "whisper-tiny", "precision": "int8 static", "size_mb": 8.8},
+    "indic-smart-turn-tiny-fp32.onnx": {"encoder": "whisper-tiny", "precision": "fp32", "size_mb": 32}
+  }
+}

indic-smart-turn-base-fp32.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5f42d53537b8622f4c4916903da2b71f4b50398cc06fd01fdf7522022f817b9d
+size 81328960

indic-smart-turn-base-int8.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:98ef8f8e2185ab337bb622cf60cf353c5235d9db91182dd5c56fe7fbda8dc172
+size 23960494

indic-smart-turn-tiny-fp32.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4b46c144a7f290388087112934d685460a2afa0b767a5fa5e0a621b6313bd904
+size 32070911

indic-smart-turn-tiny-int8.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1cfda3c0935de4eab03e2e02b7643d1382cf98c08245e12f9fb091e10691b3dd
+size 8785144

int8_cpu_dependence.md

@@ -0,0 +1,26 @@
+# int8 results depend on the CPU architecture
+
+The same ONNX files were scored on the same test clips on two machines: an x86 cloud server (AMD EPYC 7543) and a MacBook (Apple M4 Pro, Arm). Four languages, test split, one evaluator.
+
+| Language | Model | Server (x86) | MacBook (Arm) | Difference |
+|---|---|---:|---:|---:|
+| Kannada | Smart Turn v3.2 fp32 | 56.6 / 0.795 | 56.6 / 0.795 | 0 |
+| Kannada | Indic tiny fp32 | 82.3 / 0.876 | 82.3 / 0.876 | 0 |
+| Kannada | Smart Turn v3.2 int8 (static) | 77.7 / 0.770 | 74.0 / 0.783 | -3.7 |
+| Kannada | Indic tiny int8 (static) | 82.9 / 0.878 | 84.2 / 0.895 | +1.3 |
+| Malayalam | Smart Turn v3.2 int8 (static) | 66.9 / 0.657 | 63.8 / 0.664 | -3.1 |
+| Malayalam | Indic tiny int8 (static) | 80.9 / 0.823 | 79.4 / 0.830 | -1.5 |
+| Telugu | Smart Turn v3.2 int8 (static) | 73.4 / 0.750 | 70.9 / 0.766 | -2.5 |
+| Telugu | Indic tiny int8 (static) | 85.1 / 0.898 | 87.9 / 0.900 | +2.8 |
+| Assamese | Smart Turn v3.2 int8 (static) | 67.8 / 0.696 | 64.9 / 0.691 | -2.9 |
+| Assamese | Indic tiny int8 (static) | 80.4 / 0.837 | 81.5 / 0.850 | +1.1 |
+
+Every fp32 model gives identical accuracy and AUC on both machines, so the audio decoding and feature extraction are identical. Only the int8 files move. ONNX Runtime uses different int8 kernels on x86 and Arm, with different rounding and saturation, so a quantised model is not the same function on the two architectures. Statically quantised files move by up to 3.7 points on a single language. Over the whole 20,431-clip public test, Smart Turn v3.2 int8 scores 81.7 / 0.894 on the server and 81.9 / 0.902 on the MacBook.
+
+What this means for reading the results:
+
+- Compare int8 numbers only when both models were scored on the same machine. Every int8 chart in the README pairs models scored on one machine.
+- fp32 numbers can be compared across machines. int8 numbers can differ by about 3 points per language and 1 point overall.
+- On Apple Silicon use the fp32 files: they are faster there, and their results do not depend on the CPU.
+
+Raw run for the MacBook side: `python -m indic_turn.eval` on `data/built/{kan,mal,tel,asm}.parquet`, split test.

pause-test.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f8df26a4416bb37eaea4d7db30abdb967898feeac447c2ce790ab35b4db84305
+size 2949867