File size: 3,040 Bytes
8976c3c | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 | ---
license: other
license_name: openmdw-1.1
base_model: nvidia/Nemotron-3-Diarization
library_name: coreml
pipeline_tag: voice-activity-detection
tags:
- speaker-diarization
- streaming-sortformer
- coreml
- apple-silicon
---
# Nemotron 3 Diarization — Core ML
Core ML export of the final
[`nvidia/Nemotron-3-Diarization`](https://huggingface.co/nvidia/Nemotron-3-Diarization)
checkpoint (revision `98fcee8e866bb534eee85d1eb3817170700ae717`), for on-device
speaker diarization on macOS 14+ / iOS 17+.
The license terms of the source checkpoint apply to this derivative.
## Files
| File | Purpose |
|---|---|
| `nemotron_3_diarization.mlpackage` | Streaming Sortformer step: pre-encoder, Fast-Conformer, transformer, speaker heads |
| `learnable_sil_emb.f32` | 512 raw little-endian float32 values: the trained silence embedding for the speaker cache |
| `config.json` | Streaming geometry, cache and post-processing settings, I/O shapes, SHA-256 of every file |
| `conversion/` | Scripts that produced the package |
## Precision: fp16 storage, fp32 compute
Weights are stored as fp16 and widened to fp32 at load time (`constexpr_cast`),
so every op computes in fp32. The package stays at fp16 size (~199 MB).
- The checkpoint is bf16, which fp16 represents almost exactly (0.005% of
values differ, by less than 3e-8).
- A plain fp16-compute export drifts on long audio: small per-chunk rounding
feeds back through the speaker cache and compounds. On a 36-minute meeting
it matched the reference on only 98.4–98.9% of frames and added a speaker.
This happened on GPU, CPU and Neural Engine alike.
## Validation
Frame-level agreement with NVIDIA's NeMo reference, same streaming loop and
post-processing, on real meeting recordings:
| Audio | NeMo speakers | Core ML speakers | Frame agreement |
|---|---|---|---|
| 70 s, one microphone | 2 | 2 | 100.00% |
| 36 min, microphone track | 4 | 4 | 99.94% |
| 36 min, application-audio track | 5 | 5 | 100.00% |
Speed: ~21 ms per 27.2 s chunk on Apple silicon (about 2 s for a 36-minute track).
## Interface
One call processes one chunk; the caller owns the speaker cache and FIFO
update between calls (NeMo `streaming_update`).
| Input | Shape | Type |
|---|---|---|
| `chunk` | 1 × 3040 × 128 | float32 log-mel, 10 ms hop |
| `chunk_lengths` | 1 | int32 |
| `spkcache` | 1 × 264 × 512 | float32 |
| `spkcache_lengths` | 1 | int32 |
| `fifo` | 1 × 40 × 512 | float32 |
| `fifo_lengths` | 1 | int32 |
| Output | Shape |
|---|---|
| `spkcache_fifo_chunk_preds` | 1 × 684 × 8 |
| `chunk_pre_encode_embs` | 1 × 380 × 512 |
| `chunk_pre_encode_lengths` | 1 |
## Required runtime settings
These must match training, or the model splits one voice across several
speaker slots:
- `spkcache_sil_frames_per_spk = 1`
- Fill silence slots in the speaker cache with `learnable_sil_emb.f32`,
not a running mean of silent frames.
- Offline profile: `chunk_len 340`, `chunk_right_context 40`,
`fifo_len 40`, `spkcache_len 264`, `spkcache_update_period 300`.
|