Blame for README.md

---
license: openmdw-1.1
base_model: nvidia/Nemotron-3-Diarization
library_name: fluidaudio
pipeline_tag: voice-activity-detection
tags:
  - speaker-diarization
  - streaming
  - coreml
  - apple-silicon
  - ane
  - sortformer
---

# Nemotron 3 Diarization — CoreML for Apple Silicon

CoreML conversion of NVIDIA's [Nemotron 3 Diarization](https://huggingface.co/nvidia/Nemotron-3-Diarization)
(8-speaker streaming Sortformer, 100M params, 31-layer RoPE Transformer, 10 ms output
resolution), optimized for on-device inference on Apple platforms via
[FluidAudio](https://github.com/FluidInference/FluidAudio).

Runs fully on-device — Neural Engine or GPU — on macOS 14+ / iOS 17+.

## Highlights (M5 Pro)

- **Reproduces NVIDIA's model-card accuracy on-device**: AMI MHM `offline` 9.47 DER
  (card 9.25), `low` 9.75 (card 9.48) — same forced-alignment references, collar 0.
  The ~0.25 gap is the fp16 CoreML runtime against NVIDIA's bf16 GPU numbers, and it
  is consistent across presets rather than drifting.
- **`fast128` is the accuracy pick**: 9.36 DER at 546x wall real-time, and it counts
  speakers correctly on all 16 AMI test meetings — better than any row on NVIDIA's
  card, which tops out at 87.5% speaker-counting accuracy.
- **8-speaker streaming down to 1.04 s latency**, the card's own reference config.
- **100% ANE-resident W8A8 split builds at half the weight footprint**:
  `c128-split-w8a8` gives up 0.27 DER for 95 MB and keeps 16/16 speaker counting.

## Which model should I use?

| Model | Size | Audio/call | Latency | DER¹ | SCA¹ | Wall RTFx | Pros | Cons |
|---|---|---|---|---|---|---|---|---|
| `fast128` | 190 MB | 10.24 s | 10.56 s | **9.36** | **100.0** | 546x | Best accuracy and speaker counting of the set | 10.5 s latency — near-live only |
| `offline` | 190 MB | 27.2 s | 30.4 s | 9.47 | 87.5 | **904x** | Fastest wall clock; the card's batch config | No ANE (compiler limit); 30 s latency |
| `fast32` | 190 MB | 2.56 s | 2.88 s | 9.53 | 93.8 | 179x | **Recommended default** — near-`offline` quality at conversational latency | 2.88 s latency; not for live captions |
| `c128-split-w8a8` | **95 MB** | 10.24 s | 10.56 s | 9.63 | **100.0** | 364x | Half size, 100% ANE, counting intact — the batch/iOS pick | Split mode needs host pre-encode (FluidAudio handles it) |
| `low` | 190 MB | 0.72 s | 1.04 s | 9.75 | 75.0 | 31x | Lowest latency with the large FIFO; the card's streaming config | Heaviest per second of audio; weaker speaker counting |
| `fast32-split-w8a8` | **95 MB** | 2.56 s | 2.88 s | 9.76 | 75.0 | 185x | Half size at conversational latency | Speaker counting drops 18.8 pts vs fp16 `fast32` — see note |
| `fast` | 190 MB | 0.72 s | 1.04 s | 10.07 | 68.8 | 44x | ~1.4x cheaper than `low` at the same latency | Worst DER and counting of the set |

¹ AMI MHM test, 16 meetings, forced-alignment references
([nttcslab-sp/diar-forced-alignment](https://github.com/nttcslab-sp/diar-forced-alignment)),
collar 0, overlap included — the protocol NVIDIA's card uses. SCA = speaker-counting
accuracy (fraction of meetings whose speaker count is exactly right). NVIDIA's published
AMI MHM numbers under this protocol: 9.25 DER / 87.5 SCA offline, 9.48 / 81.25 at 1.04 s.
Wall RTFx is single-stream on an M5 Pro MacBook, ANE route for the split builds.

> **Window size drives speaker counting, not quantization.** On the 10.24 s window the
> W8A8 build matches fp16 at 16/16 meetings for 0.27 DER. On the 2.56 s window it loses
> 18.8 points of counting accuracy. Short windows are the fragile ones in fp16 too —
> `low` and `fast` sit at 75.0 and 68.8 on a 0.72 s window. If speaker count matters more
> than latency, prefer a longer window.

## Presets

All presets share one checkpoint; they differ only in streaming shape. Latency =
(chunk + right-context) x 80 ms. `Packed T` is the transformer sequence length
(speaker cache + FIFO + window) and is what sets cost per call. DER/SCA: AMI MHM test,
16 meetings, forced-alignment references, collar 0.

### Monolithic (single model file, simplest integration)

| Preset | Chunk | RC | FIFO | Spk cache | Packed T | Latency | DER | SCA | Wall RTFx |
|---|---|---|---|---|---|---|---|---|---|
| `offline` | 340 | 40 | 40 | 264 | 684 | 30.40 s | 9.47 | 87.5 | **904x** |
| `fast128` | 128 | 4 | 40 | 264 | 436 | 10.56 s | **9.36** | **100.0** | 546x |
| `fast32` | 32 | 4 | 40 | 264 | 340 | 2.88 s | 9.53 | 93.8 | 179x |
| `low` | 9 | 4 | 264 | 264 | 541 | 1.04 s | 9.75 | 75.0 | 31x |
| `fast` | 9 | 4 | 40 | 264 | 317 | 1.04 s | 10.07 | 68.8 | 44x |

`low` and `fast` run the same 0.72 s window at the same latency; the only difference is
the FIFO (264 vs 40), which takes the sequence from 317 to 541. That buys `low` 0.32 DER
and 6.2 points of speaker counting, and costs roughly a third of the throughput.

Other configurations (sub-second latency tiers, intermediate chunk sizes, weight-only
int8) were built and benchmarked but are not published — they are dominated by the
presets above. Open an issue if you need one; regeneration is scripted.

### Split-graph (host pre-encode; 100% ANE-resident CoreML subgraph)

Feature stacking, the 1024->512 projection (`pre_encode_proj_t.bin`), state packing,
and masks run host-side; the model is the pure-fp transformer+head. FluidAudio handles
all of this via the `-split` presets.

| Preset | Packed T | Latency | DER | SCA | Wall RTFx | Weights |
|---|---|---|---|---|---|---|
| `c128-split-w8a8` | 436 | 10.56 s | 9.63 | **100.0** | 364x | **95 MB** |
| `fast32-split-w8a8` | 340 | 2.88 s | 9.76 | 75.0 | 185x | **95 MB** |

## Usage (FluidAudio, Swift)

```swift
import FluidAudio

let config = Nemotron3Config.fast32  // or .preset(named: "c128-split-w8a8")!
let models = try await Nemotron3Models.load(
    config: config,
    directory: modelsDirectoryURL  // this repo's files
)
let diarizer = Nemotron3Diarizer(config: config, models: models)

let (probs, frames) = try diarizer.processComplete(audioSamples)  // 16 kHz mono
let segments = Nemotron3Diarizer.segments(probabilities: probs, frameCount: frames)
// -> arrival-ordered speaker segments at 10 ms resolution, up to 8 speakers
```

Optional VAD gating for sparse audio (2x wall speedup at 55% speech density):

```swift
let (probs, frames) = try diarizer.processComplete(audioSamples, speechMask: mask)
```

## Files

- `monolithic/` — one `.mlmodelc` per preset (mel features in, predictions out)
- `monolithic/v2/` — same presets re-exported with fp16 inputs/outputs and gather-free state
  packing, so they compile for the ANE on M3-class chips ([FluidAudio #951](https://github.com/FluidInference/FluidAudio/issues/951));
  same checkpoint and weights, DER-neutral on AMI. FluidAudio v0.17.3+ loads these; the
  originals in `monolithic/` are kept for older releases.
- `split/` — W8A8 transformer graphs (`packed`/`attn_bias`/`output_mask` inputs)
- `learnable_sil_emb.bin` — learned silence embedding, 512 x fp32 (cache compression)
- `pre_encode_proj_t.bin` — FeatureStacking projection W^T, 1024x512 fp32 (split mode)

All models: fixed shapes, fp16 weights unless noted, `minimum_deployment_target` iOS 17 /
macOS 14. Conversion verified against the NeMo reference across four state configurations
per preset (cold start, half-full cache, steady state, partial final chunk): the export
wrapper is exact in torch (0.0), and the CoreML fp16 runtime stays within 1.9e-4 on
speaker predictions.

## Attribution & license

Converted from [nvidia/Nemotron-3-Diarization](https://huggingface.co/nvidia/Nemotron-3-Diarization)
(`Nemotron-3-Diarization.nemo`, sha256 `867c53f5…`), released by NVIDIA under the
OpenMDW License v1.1, which permits commercial use. The converted weights carry the
same licence.
Conversion pipeline, Swift runtime, and benchmarks by
[Fluid Inference](https://huggingface.co/FluidInference).

Reference: [Streaming Sortformer](https://arxiv.org/abs/2507.18446).