Add monolithic/v2: fp16 I/O, gather-free presets that compile on M3 ANE (FluidAudio #951)

1b0b133 verified about 1 month ago

7.95 kB

---
license: openmdw-1.1
base_model: nvidia/Nemotron-3-Diarization
library_name: fluidaudio
pipeline_tag: voice-activity-detection
tags:
  - speaker-diarization
  - streaming
  - coreml
  - apple-silicon
  - ane
  - sortformer
---
# Nemotron 3 Diarization — CoreML for Apple Silicon

CoreML conversion of NVIDIA's [Nemotron 3 Diarization](https://huggingface.co/nvidia/Nemotron-3-Diarization)
(8-speaker streaming Sortformer, 100M params, 31-layer RoPE Transformer, 10 ms output
resolution), optimized for on-device inference on Apple platforms via
[FluidAudio](https://github.com/FluidInference/FluidAudio).

Runs fully on-device — Neural Engine or GPU — on macOS 14+ / iOS 17+.

## Highlights (M5 Pro)

- **Reproduces NVIDIA's model-card accuracy on-device**: AMI MHM `offline` 9.47 DER
  (card 9.25), `low` 9.75 (card 9.48) — same forced-alignment references, collar 0.
  The ~0.25 gap is the fp16 CoreML runtime against NVIDIA's bf16 GPU numbers, and it
  is consistent across presets rather than drifting.
- **`fast128` is the accuracy pick**: 9.36 DER at 546x wall real-time, and it counts
  speakers correctly on all 16 AMI test meetings — better than any row on NVIDIA's
  card, which tops out at 87.5% speaker-counting accuracy.
- **8-speaker streaming down to 1.04 s latency**, the card's own reference config.
- **100% ANE-resident W8A8 split builds at half the weight footprint**:
  `c128-split-w8a8` gives up 0.27 DER for 95 MB and keeps 16/16 speaker counting.

## Which model should I use?

| Model | Size | Audio/call | Latency | DER¹ | SCA¹ | Wall RTFx | Pros | Cons |
|---|---|---|---|---|---|---|---|---|
| `fast128` | 190 MB | 10.24 s | 10.56 s | **9.36** | **100.0** | 546x | Best accuracy and speaker counting of the set | 10.5 s latency — near-live only |
| `offline` | 190 MB | 27.2 s | 30.4 s | 9.47 | 87.5 | **904x** | Fastest wall clock; the card's batch config | No ANE (compiler limit); 30 s latency |
| `fast32` | 190 MB | 2.56 s | 2.88 s | 9.53 | 93.8 | 179x | **Recommended default** — near-`offline` quality at conversational latency | 2.88 s latency; not for live captions |
| `c128-split-w8a8` | **95 MB** | 10.24 s | 10.56 s | 9.63 | **100.0** | 364x | Half size, 100% ANE, counting intact — the batch/iOS pick | Split mode needs host pre-encode (FluidAudio handles it) |
| `low` | 190 MB | 0.72 s | 1.04 s | 9.75 | 75.0 | 31x | Lowest latency with the large FIFO; the card's streaming config | Heaviest per second of audio; weaker speaker counting |
| `fast32-split-w8a8` | **95 MB** | 2.56 s | 2.88 s | 9.76 | 75.0 | 185x | Half size at conversational latency | Speaker counting drops 18.8 pts vs fp16 `fast32` — see note |
| `fast` | 190 MB | 0.72 s | 1.04 s | 10.07 | 68.8 | 44x | ~1.4x cheaper than `low` at the same latency | Worst DER and counting of the set |

¹ AMI MHM test, 16 meetings, forced-alignment references
([nttcslab-sp/diar-forced-alignment](https://github.com/nttcslab-sp/diar-forced-alignment)),
collar 0, overlap included — the protocol NVIDIA's card uses. SCA = speaker-counting
accuracy (fraction of meetings whose speaker count is exactly right). NVIDIA's published
AMI MHM numbers under this protocol: 9.25 DER / 87.5 SCA offline, 9.48 / 81.25 at 1.04 s.
Wall RTFx is single-stream on an M5 Pro MacBook, ANE route for the split builds.

> **Window size drives speaker counting, not quantization.** On the 10.24 s window the
> W8A8 build matches fp16 at 16/16 meetings for 0.27 DER. On the 2.56 s window it loses
> 18.8 points of counting accuracy. Short windows are the fragile ones in fp16 too —
> `low` and `fast` sit at 75.0 and 68.8 on a 0.72 s window. If speaker count matters more
> than latency, prefer a longer window.

## Presets

All presets share one checkpoint; they differ only in streaming shape. Latency =
(chunk + right-context) x 80 ms. `Packed T` is the transformer sequence length
(speaker cache + FIFO + window) and is what sets cost per call. DER/SCA: AMI MHM test,
16 meetings, forced-alignment references, collar 0.

### Monolithic (single model file, simplest integration)

| Preset | Chunk | RC | FIFO | Spk cache | Packed T | Latency | DER | SCA | Wall RTFx |
|---|---|---|---|---|---|---|---|---|---|
| `offline` | 340 | 40 | 40 | 264 | 684 | 30.40 s | 9.47 | 87.5 | **904x** |
| `fast128` | 128 | 4 | 40 | 264 | 436 | 10.56 s | **9.36** | **100.0** | 546x |
| `fast32` | 32 | 4 | 40 | 264 | 340 | 2.88 s | 9.53 | 93.8 | 179x |
| `low` | 9 | 4 | 264 | 264 | 541 | 1.04 s | 9.75 | 75.0 | 31x |
| `fast` | 9 | 4 | 40 | 264 | 317 | 1.04 s | 10.07 | 68.8 | 44x |

`low` and `fast` run the same 0.72 s window at the same latency; the only difference is
the FIFO (264 vs 40), which takes the sequence from 317 to 541. That buys `low` 0.32 DER
and 6.2 points of speaker counting, and costs roughly a third of the throughput.

Other configurations (sub-second latency tiers, intermediate chunk sizes, weight-only
int8) were built and benchmarked but are not published — they are dominated by the
presets above. Open an issue if you need one; regeneration is scripted.

### Split-graph (host pre-encode; 100% ANE-resident CoreML subgraph)

Feature stacking, the 1024->512 projection (`pre_encode_proj_t.bin`), state packing,
and masks run host-side; the model is the pure-fp transformer+head. FluidAudio handles
all of this via the `-split` presets.

| Preset | Packed T | Latency | DER | SCA | Wall RTFx | Weights |
|---|---|---|---|---|---|---|
| `c128-split-w8a8` | 436 | 10.56 s | 9.63 | **100.0** | 364x | **95 MB** |
| `fast32-split-w8a8` | 340 | 2.88 s | 9.76 | 75.0 | 185x | **95 MB** |

## Usage (FluidAudio, Swift)

```swift
import FluidAudio
let config = Nemotron3Config.fast32  // or .preset(named: "c128-split-w8a8")!
let models = try await Nemotron3Models.load(
    config: config,
    directory: modelsDirectoryURL  // this repo's files
)
let diarizer = Nemotron3Diarizer(config: config, models: models)
let (probs, frames) = try diarizer.processComplete(audioSamples)  // 16 kHz mono
let segments = Nemotron3Diarizer.segments(probabilities: probs, frameCount: frames)
// -> arrival-ordered speaker segments at 10 ms resolution, up to 8 speakers
```

Optional VAD gating for sparse audio (2x wall speedup at 55% speech density):

```swift
let (probs, frames) = try diarizer.processComplete(audioSamples, speechMask: mask)
```

## Files

- `monolithic/` — one `.mlmodelc` per preset (mel features in, predictions out)
- `monolithic/v2/` — same presets re-exported with fp16 inputs/outputs and gather-free state
  packing, so they compile for the ANE on M3-class chips ([FluidAudio #951](https://github.com/FluidInference/FluidAudio/issues/951));
  same checkpoint and weights, DER-neutral on AMI. FluidAudio v0.17.3+ loads these; the
  originals in `monolithic/` are kept for older releases.
- `split/` — W8A8 transformer graphs (`packed`/`attn_bias`/`output_mask` inputs)
- `learnable_sil_emb.bin` — learned silence embedding, 512 x fp32 (cache compression)
- `pre_encode_proj_t.bin` — FeatureStacking projection W^T, 1024x512 fp32 (split mode)

All models: fixed shapes, fp16 weights unless noted, `minimum_deployment_target` iOS 17 /
macOS 14. Conversion verified against the NeMo reference across four state configurations
per preset (cold start, half-full cache, steady state, partial final chunk): the export
wrapper is exact in torch (0.0), and the CoreML fp16 runtime stays within 1.9e-4 on
speaker predictions.

## Attribution & license

Converted from [nvidia/Nemotron-3-Diarization](https://huggingface.co/nvidia/Nemotron-3-Diarization)
(`Nemotron-3-Diarization.nemo`, sha256 `867c53f5…`), released by NVIDIA under the
OpenMDW License v1.1, which permits commercial use. The converted weights carry the
same licence.
Conversion pipeline, Swift runtime, and benchmarks by
[Fluid Inference](https://huggingface.co/FluidInference).

Reference: [Streaming Sortformer](https://arxiv.org/abs/2507.18446).

	---
	license: openmdw-1.1
	base_model: nvidia/Nemotron-3-Diarization
	library_name: fluidaudio
	pipeline_tag: voice-activity-detection
	tags:
	- speaker-diarization
	- streaming
	- coreml
	- apple-silicon
	- ane
	- sortformer
	---

	# Nemotron 3 Diarization — CoreML for Apple Silicon

	CoreML conversion of NVIDIA's [Nemotron 3 Diarization](https://huggingface.co/nvidia/Nemotron-3-Diarization)
	(8-speaker streaming Sortformer, 100M params, 31-layer RoPE Transformer, 10 ms output
	resolution), optimized for on-device inference on Apple platforms via
	[FluidAudio](https://github.com/FluidInference/FluidAudio).

	Runs fully on-device — Neural Engine or GPU — on macOS 14+ / iOS 17+.

	## Highlights (M5 Pro)

	- Reproduces NVIDIA's model-card accuracy on-device: AMI MHM `offline` 9.47 DER
	(card 9.25), `low` 9.75 (card 9.48) — same forced-alignment references, collar 0.
	The ~0.25 gap is the fp16 CoreML runtime against NVIDIA's bf16 GPU numbers, and it
	is consistent across presets rather than drifting.
	- `fast128` is the accuracy pick: 9.36 DER at 546x wall real-time, and it counts
	speakers correctly on all 16 AMI test meetings — better than any row on NVIDIA's
	card, which tops out at 87.5% speaker-counting accuracy.
	- 8-speaker streaming down to 1.04 s latency, the card's own reference config.
	- 100% ANE-resident W8A8 split builds at half the weight footprint:
	`c128-split-w8a8` gives up 0.27 DER for 95 MB and keeps 16/16 speaker counting.

	## Which model should I use?

	\| Model \| Size \| Audio/call \| Latency \| DER¹ \| SCA¹ \| Wall RTFx \| Pros \| Cons \|
	\|---\|---\|---\|---\|---\|---\|---\|---\|---\|
	\| `fast128` \| 190 MB \| 10.24 s \| 10.56 s \| 9.36 \| 100.0 \| 546x \| Best accuracy and speaker counting of the set \| 10.5 s latency — near-live only \|
	\| `offline` \| 190 MB \| 27.2 s \| 30.4 s \| 9.47 \| 87.5 \| 904x \| Fastest wall clock; the card's batch config \| No ANE (compiler limit); 30 s latency \|
	\| `fast32` \| 190 MB \| 2.56 s \| 2.88 s \| 9.53 \| 93.8 \| 179x \| Recommended default — near-`offline` quality at conversational latency \| 2.88 s latency; not for live captions \|
	\| `c128-split-w8a8` \| 95 MB \| 10.24 s \| 10.56 s \| 9.63 \| 100.0 \| 364x \| Half size, 100% ANE, counting intact — the batch/iOS pick \| Split mode needs host pre-encode (FluidAudio handles it) \|
	\| `low` \| 190 MB \| 0.72 s \| 1.04 s \| 9.75 \| 75.0 \| 31x \| Lowest latency with the large FIFO; the card's streaming config \| Heaviest per second of audio; weaker speaker counting \|
	\| `fast32-split-w8a8` \| 95 MB \| 2.56 s \| 2.88 s \| 9.76 \| 75.0 \| 185x \| Half size at conversational latency \| Speaker counting drops 18.8 pts vs fp16 `fast32` — see note \|
	\| `fast` \| 190 MB \| 0.72 s \| 1.04 s \| 10.07 \| 68.8 \| 44x \| ~1.4x cheaper than `low` at the same latency \| Worst DER and counting of the set \|

	¹ AMI MHM test, 16 meetings, forced-alignment references
	([nttcslab-sp/diar-forced-alignment](https://github.com/nttcslab-sp/diar-forced-alignment)),
	collar 0, overlap included — the protocol NVIDIA's card uses. SCA = speaker-counting
	accuracy (fraction of meetings whose speaker count is exactly right). NVIDIA's published
	AMI MHM numbers under this protocol: 9.25 DER / 87.5 SCA offline, 9.48 / 81.25 at 1.04 s.
	Wall RTFx is single-stream on an M5 Pro MacBook, ANE route for the split builds.

	> Window size drives speaker counting, not quantization. On the 10.24 s window the
	> W8A8 build matches fp16 at 16/16 meetings for 0.27 DER. On the 2.56 s window it loses
	> 18.8 points of counting accuracy. Short windows are the fragile ones in fp16 too —
	> `low` and `fast` sit at 75.0 and 68.8 on a 0.72 s window. If speaker count matters more
	> than latency, prefer a longer window.

	## Presets

	All presets share one checkpoint; they differ only in streaming shape. Latency =
	(chunk + right-context) x 80 ms. `Packed T` is the transformer sequence length
	(speaker cache + FIFO + window) and is what sets cost per call. DER/SCA: AMI MHM test,
	16 meetings, forced-alignment references, collar 0.

	### Monolithic (single model file, simplest integration)

	\| Preset \| Chunk \| RC \| FIFO \| Spk cache \| Packed T \| Latency \| DER \| SCA \| Wall RTFx \|
	\|---\|---\|---\|---\|---\|---\|---\|---\|---\|---\|
	\| `offline` \| 340 \| 40 \| 40 \| 264 \| 684 \| 30.40 s \| 9.47 \| 87.5 \| 904x \|
	\| `fast128` \| 128 \| 4 \| 40 \| 264 \| 436 \| 10.56 s \| 9.36 \| 100.0 \| 546x \|
	\| `fast32` \| 32 \| 4 \| 40 \| 264 \| 340 \| 2.88 s \| 9.53 \| 93.8 \| 179x \|
	\| `low` \| 9 \| 4 \| 264 \| 264 \| 541 \| 1.04 s \| 9.75 \| 75.0 \| 31x \|
	\| `fast` \| 9 \| 4 \| 40 \| 264 \| 317 \| 1.04 s \| 10.07 \| 68.8 \| 44x \|

	`low` and `fast` run the same 0.72 s window at the same latency; the only difference is
	the FIFO (264 vs 40), which takes the sequence from 317 to 541. That buys `low` 0.32 DER
	and 6.2 points of speaker counting, and costs roughly a third of the throughput.

	Other configurations (sub-second latency tiers, intermediate chunk sizes, weight-only
	int8) were built and benchmarked but are not published — they are dominated by the
	presets above. Open an issue if you need one; regeneration is scripted.

	### Split-graph (host pre-encode; 100% ANE-resident CoreML subgraph)

	Feature stacking, the 1024->512 projection (`pre_encode_proj_t.bin`), state packing,
	and masks run host-side; the model is the pure-fp transformer+head. FluidAudio handles
	all of this via the `-split` presets.

	\| Preset \| Packed T \| Latency \| DER \| SCA \| Wall RTFx \| Weights \|
	\|---\|---\|---\|---\|---\|---\|---\|
	\| `c128-split-w8a8` \| 436 \| 10.56 s \| 9.63 \| 100.0 \| 364x \| 95 MB \|
	\| `fast32-split-w8a8` \| 340 \| 2.88 s \| 9.76 \| 75.0 \| 185x \| 95 MB \|

	## Usage (FluidAudio, Swift)

	```swift
	import FluidAudio

	let config = Nemotron3Config.fast32 // or .preset(named: "c128-split-w8a8")!
	let models = try await Nemotron3Models.load(
	config: config,
	directory: modelsDirectoryURL // this repo's files
	)
	let diarizer = Nemotron3Diarizer(config: config, models: models)

	let (probs, frames) = try diarizer.processComplete(audioSamples) // 16 kHz mono
	let segments = Nemotron3Diarizer.segments(probabilities: probs, frameCount: frames)
	// -> arrival-ordered speaker segments at 10 ms resolution, up to 8 speakers
	```

	Optional VAD gating for sparse audio (2x wall speedup at 55% speech density):

	```swift
	let (probs, frames) = try diarizer.processComplete(audioSamples, speechMask: mask)
	```

	## Files

	- `monolithic/` — one `.mlmodelc` per preset (mel features in, predictions out)
	- `monolithic/v2/` — same presets re-exported with fp16 inputs/outputs and gather-free state
	packing, so they compile for the ANE on M3-class chips ([FluidAudio #951](https://github.com/FluidInference/FluidAudio/issues/951));
	same checkpoint and weights, DER-neutral on AMI. FluidAudio v0.17.3+ loads these; the
	originals in `monolithic/` are kept for older releases.
	- `split/` — W8A8 transformer graphs (`packed`/`attn_bias`/`output_mask` inputs)
	- `learnable_sil_emb.bin` — learned silence embedding, 512 x fp32 (cache compression)
	- `pre_encode_proj_t.bin` — FeatureStacking projection W^T, 1024x512 fp32 (split mode)

	All models: fixed shapes, fp16 weights unless noted, `minimum_deployment_target` iOS 17 /
	macOS 14. Conversion verified against the NeMo reference across four state configurations
	per preset (cold start, half-full cache, steady state, partial final chunk): the export
	wrapper is exact in torch (0.0), and the CoreML fp16 runtime stays within 1.9e-4 on
	speaker predictions.

	## Attribution & license

	Converted from [nvidia/Nemotron-3-Diarization](https://huggingface.co/nvidia/Nemotron-3-Diarization)
	(`Nemotron-3-Diarization.nemo`, sha256 `867c53f5…`), released by NVIDIA under the
	OpenMDW License v1.1, which permits commercial use. The converted weights carry the
	same licence.
	Conversion pipeline, Swift runtime, and benchmarks by
	[Fluid Inference](https://huggingface.co/FluidInference).

	Reference: [Streaming Sortformer](https://arxiv.org/abs/2507.18446).