s1lvrn0/whiscribe-diarization

Speaker diarization models for Whiscribe (ONNX)

The two ONNX models Whiscribe, an Android app that transcribes audio on the phone, uses to find who speaks when. They run on ONNX Runtime's CPU provider, with a small pipeline of the app's own around them.

File	Size	SHA-256	What it does
pyannote-segmentation-3.0.onnx	5,992,913 bytes	220ad67ca923bef2fa91f2390c786097bf305bceb5e261d4af67b38e938e1079	Who talks in each 10 s window, up to three speakers (powerset output)
redimnet2-b3-vox2-lm.onnx	18,041,088 bytes	4778b1e84c49d7e3dc0ee5caf6aef69a357f42a8f73d38b42cb39bf39024e452	A voice as 192 numbers (speaker embedding), from 16 kHz audio

Segmentation: input x float32 [batch, 1, 160000] (10 s at 16 kHz), output y [batch, 589, 7]: per frame, the scores of the 7 powerset classes (nobody, speaker 1, 2, 3, 1+2, 1+3, 2+3).

Embedding: input wav float32 [1, samples] at 16 kHz, mono, in [-1, 1], any length (the app uses 0.5 s or more); output emb [1, 192]. Compare two by cosine similarity.

NPU builds of ReDimNet2-B3 (Snapdragon)

The same ReDimNet2-B3, exported at one fixed length (export_redimnet2_onnx.py --seconds 3: input wav [1, 48000], output emb [1, 192]) and compiled on Qualcomm AI Hub with QAIRT 2.50 into QNN context binaries for each chip's Hexagon NPU (fp16). They only run with Qualcomm's QNN runtime on that chip. For the NPU, the export writes one layer (stage4.2, a depthwise convolution 1 x 4 with a stride of 4, which took 73% of the NPU's time) as the same arithmetic in plain multiplies and a sum; its output matches PyTorch's (cosine 1.0). Whiscribe repeats a shorter piece to fill 3 s, cuts a longer one into 3 s chunks and averages their unit vectors; on AMI meetings that scores the same as any length on the CPU.

File	Chip	Size	SHA-256
redimnet2-b3-npu-sm8850.bin	Snapdragon 8 Elite Gen 5 (Hexagon V81)	21,071,408 bytes	50391c0973a556aca2eca5ff94497101b78efa6186833a656c7621ab8d8df936
redimnet2-b3-npu-sm8750.bin	Snapdragon 8 Elite (Hexagon V79)	21,022,256 bytes	a33b49ac4816f09b34e10606fb3065b8ab2a35ca01358c9f348a5950be6ea7e9
redimnet2-b3-npu-sm8650.bin	Snapdragon 8 Gen 3 (Hexagon V75)	21,063,216 bytes	4e0a116e8417b62ef7e9d03493e34d67d2fba9bad781060ac37f23267a275244
redimnet2-b3-npu-sm8550.bin	Snapdragon 8 Gen 2 (Hexagon V73)	21,030,448 bytes	08929917f38c012c40b643c6338b921b90bad85666a98ecb583589deea2eda89

How Whiscribe uses them

pyannote segmentation on 10 s windows every 2.5 s; one embedding per window speaker, from the audio where only they talk; average-linkage clustering on cosine distance, cut at 0.6; small clusters merged into the nearest bigger one in time. Measured on AMI meetings and made-up conversations of real voices in eight languages: see the app's design notes.

Credits and licence

• pyannote segmentation 3.0 by pyannote (CNRS), MIT (LICENSE-pyannote). This is the ONNX conversion published by k2-fsa/sherpa-onnx (sherpa-onnx-pyannote-segmentation-3-0.tar.bz2, its model.onnx), unchanged.
• ReDimNet2-B3 by Palabra.ai, MIT (LICENSE-redimnet2): the weights b3-vox2-lm.pt from its v1.0.0 release (SHA-256 02d14efdc184573873b5c3f62bea847fcdb6919fbba19a8da7af7c1ba36b76a2), trained on VoxCeleb2 only, exported to ONNX at opset 17 with export_redimnet2_onnx.py (repository at commit c5bbe0b76e37df698c403f8844e41304ceab6307, PyTorch's TorchScript exporter). Its embeddings match PyTorch's (cosine 1.000000 on 0.5–10 s of audio). Nothing was retrained.

This repository is MIT-licensed as well. It isn't affiliated with or endorsed by pyannote, CNRS, k2-fsa or Palabra.ai.

About the training data: VoxCeleb2's audio comes from YouTube interviews; its metadata is CC BY-SA 4.0 and the videos' copyright stays with their owners, the same lineage as nearly every English speaker model. ReDimNet2's weights trained with VoxBlink2 are CC BY-NC-SA and are not used here.

python export_redimnet2_onnx.py path/to/redimnet2 b3-vox2-lm.pt redimnet2-b3-vox2-lm.onnx
# the NPU builds' input, before compiling
python export_redimnet2_onnx.py path/to/redimnet2 b3-vox2-lm.pt redimnet2-b3-3s.onnx --seconds 3