File size: 4,432 Bytes
ac2dd36 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 | ---
license: cc-by-4.0
language:
- en
tags:
- speaker-diarization
- coreml
- apple-silicon
- neural-engine
- sortformer
- streaming
datasets:
- voxconverse
base_model: nvidia/diar_streaming_sortformer_4spk-v2.1
library_name: coreml
pipeline_tag: audio-classification
---
# Sortformer Diarization (CoreML)
CoreML port of [NVIDIA Sortformer](https://arxiv.org/abs/2409.06656) for end-to-end speaker diarization on Apple Silicon. Runs on the Neural Engine. No separate embedding extraction or clustering — the model directly predicts per-frame speaker activity for up to 4 speakers, with streaming state (speaker cache + FIFO) carried across calls.
## Model
| | |
|---|---|
| Architecture | Sortformer (Sort Loss + 17-layer FastConformer + 18-layer Transformer) |
| Base model | [`nvidia/diar_streaming_sortformer_4spk-v2.1`](https://huggingface.co/nvidia/diar_streaming_sortformer_4spk-v2.1) (117M params) |
| Precision | FP16 compute, FP32 I/O boundaries |
| Sample rate | 16 kHz, 128 mel bins (n_fft 400, hop 160) |
| Max speakers | 4 (arrival-order slots) |
| Speaker cache / FIFO | 188 / 40 encoder frames |
| Frame duration | 80 ms per prediction frame |
Three variants ship from the same checkpoint, differing only in chunk shape:
| Variant | New audio per call | Mel input | Use case |
|---|---|---|---|
| `Sortformer` (default) | ~27 s | `[1, 3048, 128]` | offline batch, highest throughput |
| `Sortformer_balanced` | ~8 s | `[1, 968, 128]` | ~3× faster first-load compile |
| `Sortformer_streaming` | 480 ms | `[1, 112, 128]` | incremental / realtime |
## Files
| File | Size | Description |
|---|---|---|
| `Sortformer.mlmodelc` / `.mlpackage` | ~230 MB | default variant (compiled / source) |
| `Sortformer_balanced.mlmodelc` / `.mlpackage` | ~230 MB | balanced variant |
| `Sortformer_streaming.mlmodelc` / `.mlpackage` | ~230 MB | streaming variant |
| `config.json`, `config_balanced.json`, `config_streaming.json` | <1 KB | chunk-shape hyperparameters per variant |
## Performance
Measured on M-series Apple Silicon (Neural Engine, warm):
| Variant | Per-call latency | Realtime factor |
|---|---|---|
| default | one call per ~27 s of audio | ~125–750× |
| balanced | one call per ~8 s | hundreds of × |
| streaming | **8.1 ms median per 480 ms step** | ~59× per step; ~39× end-to-end incremental |
The streaming export is parity-gated: driven by NeMo's own streaming feature
loader and cache-update algorithm, it matches NeMo's native
`forward_streaming` loop at **100% frame-decision agreement (MAE 0.0005)**.
On a five-file VoxConverse-dev pilot, the incremental session reaches 8.1%
DER (collar 0.25) with correct speaker counts on all files.
## Streaming interface
The streaming variant carries state through the CoreML interface; the
speaker-cache update runs host-side between calls.
**Inputs:** `chunk [1,112,128]`, `chunk_lengths [1]`, `spkcache [1,188,512]`,
`spkcache_lengths [1]`, `fifo [1,40,512]`, `fifo_lengths [1]`
**Outputs:** `speaker_preds_out [1,242,4]`,
`chunk_pre_encoder_embs_out [1,14,512]`, `chunk_pre_encoder_lengths_out [1]`
The pipeline contains two sub-models: **PreEncoder** (mel pre-encode +
state concat) and **Head** (FastConformer + Transformer + sigmoid heads).
## Usage
```swift
// Whole-buffer diarization
let diarizer = try await SortformerDiarizer.fromPretrained()
let result = diarizer.diarize(audio: samples, sampleRate: 16000)
// Incremental streaming: push PCM in any size, stable speaker slots
let session = try await SortformerStreamingSession.fromPretrained()
let snapshot = try session.push(audio: samples)
let final = try session.finish()
```
```bash
speech diarize meeting.wav --engine sortformer
```
## Source
- Upstream model: [nvidia/diar_streaming_sortformer_4spk-v2.1](https://huggingface.co/nvidia/diar_streaming_sortformer_4spk-v2.1) · [Sortformer paper](https://arxiv.org/abs/2409.06656) · [Streaming Sortformer paper](https://arxiv.org/abs/2507.18446)
- Upstream license: NVIDIA Open Model License (weights); this conversion is published under CC-BY-4.0 with attribution to NVIDIA.
## Links
- [speech-swift](https://github.com/soniqo/speech-swift) — Apple SDK
- [Docs](https://soniqo.audio/getting-started) — install and CLI docs
- [Guide](https://soniqo.audio/guides/diarize) — speaker diarization guide
- [soniqo.audio](https://soniqo.audio) — website
- [blog](https://soniqo.audio/blog) — blog
|