Smart Turn v3.2-Interview (8.0M Open-Weight Checkpoint)
Smart Turn v3.2-Interview is an open-weight 8,000,386-parameter audio-only end-of-utterance (EOU) / response-readiness detector released alongside Pikaia Interview EOU Benchmark: Evaluating Response Readiness in Bilingual Spoken Interviews.
It preserves the exact unmodified architecture, feature extractor, parameter count (8.0M), and drop-in ONNX Runtime interface of pipecat-ai/smart-turn-v3 (smart-turn-v3.2-gpu.onnx), fine-tuned for 1 epoch on 3,891 speaker-disjoint Mandarin and English spoken-interview decision points (262 distinct participants across 264 interviews).
Controlled Same-Architecture Evaluation (8.0M Audio-Only, Matched Released WAVs)
Holding the 8.0M Whisper-Tiny encoder + attention-pooling classifier completely fixed tests how far in-domain interview response_ready fine-tuning closes the compact-to-Speech-LLM gap (Holm–Bonferroni p_Holm controlled across the 8 primary comparisons):
| Evaluation Split / Subset | Upstream Smart Turn v3.2 (Zero-Shot) |
Smart Turn v3.2-Interview (dev-ranked seed29) |
3-Seed Individual Mean ± SD (seeds 7, 17, 29) |
3-Seed Ensemble |
|---|---|---|---|---|
Tier 2 test 300 ms ROC-AUC (N=1,100) |
0.8414 |
0.8989 (+0.0575, p=0.0010, p_Holm=0.0080) |
0.8918 ± 0.0071 |
0.8976 |
Tier 2 test 300 ms R@5% (N=1,100) |
41.09% |
62.07% (+20.98 pp, p=0.0010, p_Holm=0.0080) |
58.48% ± 3.25% |
63.36% |
Tier 2 Cascade R@5% (N=788) |
41.43% |
64.56% (+23.13 pp, p=0.0020, p_Holm=0.0080) |
60.18% ± 3.84% |
60.73% |
Tier 2 Cascade Wait@5% (N=788) |
713.0 ms |
555.1 ms (-157.9 ms, p=0.0040, p_Holm=0.0120) |
594.7 ± 35.4 ms |
578.9 ms |
Pre-Cutoff Strict Causal-243 AUC (N=243) |
0.6930 |
0.8012 (+0.1082, p=0.0010, p_Holm=0.0080) |
0.7593 ± 0.0393 |
0.7770 |
Dual-Human Agreed Gold AUC (N=42) |
0.8707 |
0.9705 (+0.0998, nominal p=0.0325, p_Holm=0.0650) |
0.9592 ± 0.0099 |
0.9637 |
External LiveKit eot-bench 300 ms AUC (N=1,058) |
0.7998 |
0.8391 (+0.0394, p=0.0010, p_Holm=0.0080) |
0.8425 ± 0.0054 |
0.8474 |
External LiveKit eot-bench Cascade Wait@5% |
780.4 ms |
734.5 ms (-45.9 ms, p=0.44, p_Holm=0.44) |
761.2 ± 28.0 ms |
766.2 ms |
Repository Layout
smart-turn-v3.2-interview.onnx: Drop-in FP32 ONNX checkpoint (seed29, highest Tier 1devN=100300 ms ROC-AUC of0.8892vs.0.8844[seed17] and0.8738[seed7] on exact releasedv1.0.0WAVs). Directly compatible with Pipecat / ONNX RuntimeSmart Turn v3pipelines.model.safetensors: Exact PyTorchsafetensorsstate dict (79tensors,8,000,386parameters) corresponding tosmart-turn-v3.2-interview.onnx.seeds/seed{7,17,29}/: All three random-seed checkpoints (.onnxand.safetensors) for exact reproduction of 3-seedmean ± SDand ensemble results.config.json: Architecture metadata, provenance breakdown, and unifiednextafterfrozen Tier 1devthresholds.inference.py: Minimal standalone ONNX Runtime scorer for16 kHzmono WAV files.SHA256SUMS: SHA256 digests for all repository artifacts.
Quickstart (Drop-In ONNX Runtime)
from huggingface_hub import hf_hub_download
from inference import score_onnx
onnx_path = hf_hub_download(
repo_id="pikaia-ai/smart-turn-v3.2-interview",
filename="smart-turn-v3.2-interview.onnx",
)
p_eot = score_onnx("utterance_300ms.wav", onnx_path=onnx_path)
print(f"Response-ready probability: {p_eot:.4f}")
Training Provenance, Selection History & Scope Boundaries
- Base Checkpoint: Losslessly restored from
pipecat-ai/smart-turn-v3(smart-turn-v3.2-gpu.onnx, rev.f766f81d, SHA256ab8dc64b...) with zero head reset and zero architectural changes. - Training Corpus (
N = 3,891,262Distinct Participants across264Interviews; Private Audio): Fine-tuned for1epoch (487updates;AdamW,lr=1e-5,weight_decay=0.01, gradient accumulation8, gradient norm clip1.0, within-language inverse-class-frequency binary cross-entropy) on3,891causally truncated bilingual spoken-interview decision points (2,757Mandarin [2,047EOT /710HOLD] +1,134English [773EOT /361HOLD]). Raw training audio is retained privately under participant privacy governance; verified participant/interview overlap and raw-WAV / float32 PCM SHA256 overlap with all2,338Pikaia Interview EOU Benchmark v1.0.0items and all2,116LiveKit eot-benchclips are0. - Exact Four-Part Label Provenance (
238 + 129 + 2,354 + 1,170 = 3,891): Comprises (i)238manually reviewed interview pauses, (ii)129legacy bilingual 3-LLM consensus pauses with strictly pre-cutoff causal re-judgments, (iii)2,354retained legacy bilingual 3-LLM consensus pauses where post-cutoff continuation was visible to the upstream judge (731with non-empty continuation), and (iv)1,170Mandarin pauses retained under unanimous 3-LLM voting (deepseek-v4-1-flash, native-audiogemini-3.8-flash,gpt-6.1-terra) from 300 ms-truncated audio/transcripts with pause duration in judge metadata. - Selection Chronology: Predictions in the training batch were logged across
12checkpoints (2expansion variants [N=3,891andN=4,070] ×3seeds ×{375 updates, 1 epoch}); because a pre-evaluation lock timestamp was not archived separately, corpus/epoch choice is documented retrospectively, while all12candidates consistently improve paired 600 ms Tier 2 AUC from0.8482to0.8864–0.9087(48.25%–60.07%R@5% vs.42.76%upstream). Because fine-tuning jointly updates interview domain acoustics, class weighting, andresponse_readylabels at fixed8.0Mscale, it establishes substantial practical headroom (49.5%wait-gap closure) without isolating supervision semantics alone.
Attribution & Citation
Upstream architecture and pretrained initialization originate from pipecat-ai/smart-turn-v3 (Pipecat / Daily, BSD-2-Clause).
@misc{lin2026pikaiaeou,
author = {Lin, Xuanxuan},
title = {Pikaia Interview EOU Benchmark: Evaluating Response Readiness in Bilingual Spoken Interviews},
year = {2026},
howpublished = {arXiv preprint},
url = {https://huggingface.co/datasets/pikaia-ai/interview-eou-benchmark}
}
- Downloads last month
- 43
Model tree for pikaia-ai/smart-turn-v3.2-interview
Base model
pipecat-ai/smart-turn-v3