| --- |
| license: other |
| license_name: openmdw-1.1 |
| base_model: nvidia/Nemotron-3-Diarization |
| library_name: coreml |
| pipeline_tag: voice-activity-detection |
| tags: |
| - speaker-diarization |
| - streaming-sortformer |
| - coreml |
| - apple-silicon |
| --- |
| |
| # Nemotron 3 Diarization — Core ML |
|
|
| Core ML export of the final |
| [`nvidia/Nemotron-3-Diarization`](https://huggingface.co/nvidia/Nemotron-3-Diarization) |
| checkpoint (revision `98fcee8e866bb534eee85d1eb3817170700ae717`), for on-device |
| speaker diarization on macOS 14+ / iOS 17+. |
|
|
| The license terms of the source checkpoint apply to this derivative. |
|
|
| ## Files |
|
|
| | File | Purpose | |
| |---|---| |
| | `nemotron_3_diarization.mlpackage` | Streaming Sortformer step: pre-encoder, Fast-Conformer, transformer, speaker heads | |
| | `learnable_sil_emb.f32` | 512 raw little-endian float32 values: the trained silence embedding for the speaker cache | |
| | `config.json` | Streaming geometry, cache and post-processing settings, I/O shapes, SHA-256 of every file | |
| | `conversion/` | Scripts that produced the package | |
|
|
| ## Precision: fp16 storage, fp32 compute |
|
|
| Weights are stored as fp16 and widened to fp32 at load time (`constexpr_cast`), |
| so every op computes in fp32. The package stays at fp16 size (~199 MB). |
|
|
| - The checkpoint is bf16, which fp16 represents almost exactly (0.005% of |
| values differ, by less than 3e-8). |
| - A plain fp16-compute export drifts on long audio: small per-chunk rounding |
| feeds back through the speaker cache and compounds. On a 36-minute meeting |
| it matched the reference on only 98.4–98.9% of frames and added a speaker. |
| This happened on GPU, CPU and Neural Engine alike. |
|
|
| ## Validation |
|
|
| Frame-level agreement with NVIDIA's NeMo reference, same streaming loop and |
| post-processing, on real meeting recordings: |
|
|
| | Audio | NeMo speakers | Core ML speakers | Frame agreement | |
| |---|---|---|---| |
| | 70 s, one microphone | 2 | 2 | 100.00% | |
| | 36 min, microphone track | 4 | 4 | 99.94% | |
| | 36 min, application-audio track | 5 | 5 | 100.00% | |
|
|
| Speed: ~21 ms per 27.2 s chunk on Apple silicon (about 2 s for a 36-minute track). |
|
|
| ## Interface |
|
|
| One call processes one chunk; the caller owns the speaker cache and FIFO |
| update between calls (NeMo `streaming_update`). |
|
|
| | Input | Shape | Type | |
| |---|---|---| |
| | `chunk` | 1 × 3040 × 128 | float32 log-mel, 10 ms hop | |
| | `chunk_lengths` | 1 | int32 | |
| | `spkcache` | 1 × 264 × 512 | float32 | |
| | `spkcache_lengths` | 1 | int32 | |
| | `fifo` | 1 × 40 × 512 | float32 | |
| | `fifo_lengths` | 1 | int32 | |
|
|
| | Output | Shape | |
| |---|---| |
| | `spkcache_fifo_chunk_preds` | 1 × 684 × 8 | |
| | `chunk_pre_encode_embs` | 1 × 380 × 512 | |
| | `chunk_pre_encode_lengths` | 1 | |
|
|
| ## Required runtime settings |
|
|
| These must match training, or the model splits one voice across several |
| speaker slots: |
|
|
| - `spkcache_sil_frames_per_spk = 1` |
| - Fill silence slots in the speaker cache with `learnable_sil_emb.f32`, |
| not a running mean of silent frames. |
| - Offline profile: `chunk_len 340`, `chunk_right_context 40`, |
| `fifo_len 40`, `spkcache_len 264`, `spkcache_update_period 300`. |
|
|
