| --- |
| license: cc-by-4.0 |
| base_model: nvidia/speakerverification_en_titanet_large |
| language: |
| - en |
| library_name: coreai |
| tags: |
| - coreai |
| - apple |
| - ios |
| - on-device |
| - speaker-verification |
| - speaker-recognition |
| - speaker-embedding |
| - titanet |
| - audio |
| --- |
| |
| # TitaNet-Large for Core AI |
|
|
| [NVIDIA TitaNet-Large](https://huggingface.co/nvidia/speakerverification_en_titanet_large) (23M parameters) ported |
| to Apple's **Core AI** runtime for on-device speaker verification and identification on iPhone, iPad and Mac. |
| Give it 16 kHz mono speech and it returns NeMo's 192-d speaker embedding, numerically matching the original: |
| **cosine ≥ 0.9999987 against NeMo's `get_embedding` on real recordings from 4 to 65 seconds.** |
| |
| One small bundle (47 MB) handles audio of any length: the graph is compiled for a single 5.12 s window, and a short |
| host loop sweeps that window over the clip and rebuilds the model's global statistics exactly. On an iPhone it peaks |
| at **117 MB** of memory and embeds a 65-second clip in **1.1 s** on the GPU. |
| |
| ## Results |
| |
| All comparisons are against NeMo's own embedding of the same audio (the `.nemo` checkpoint, `get_embedding`). |
| |
| | Check | Clips | Cosine to NeMo (min / mean) | Notes | |
| |---|---|---|---| |
| | WAV → Python host → Core AI (Mac, GPU) | 40 VoxCeleb1-test, 4.1–64.7 s | **0.9999987** / 0.9999993 | the whole pipeline: mel front end, windowed host, graph | |
| | NeMo's mel → Swift host → Core AI (iPhone, GPU) | same 40 | **0.9999987** / 0.9999993 | all embeddings finite | |
|
|
| Because the embeddings agree with NeMo's to within 1.3e-6 in cosine, verification behaves as the original does. |
| NVIDIA reports **0.66 % EER on VoxCeleb1 (cleaned trials)** for the base model; this port was not separately |
| re-scored on that benchmark. |
|
|
| ## Files |
|
|
| | Path | What it is | |
| |---|---| |
| | `titanet.aimodel/` | the Core AI bundle: functions `scan` and `head`, fp16 weights, fp32 compute | |
| | `export.json` | window length (512 frames), context margin (46), function signatures, precision recipe | |
| | `frontend/mel_fb.f32` | slaney mel filterbank [80, 257], float32 little-endian (the checkpoint's own buffer) | |
| | `frontend/mel_window.f32` | Hann window [400], periodic=False (the checkpoint's own buffer) | |
| | `frontend/config.json` | front-end constants: 16 kHz, n_fft 512, hop 160, win 400, 80 mels, 192-d embedding, threshold 0.7 | |
| |
| ## Input |
| |
| - **16 kHz, mono**, float samples in [-1, 1]. The model was trained on 16 kHz audio; resample anything else first. |
| - Any length from one 10 ms frame up. A few seconds of speech gives a stable embedding. |
| - The front end is NeMo's `AudioToMelSpectrogramPreprocessor` in eval mode, reproduced step for step: pre-emphasis |
| 0.97, centred 512-point STFT (hop 160, Hann 400), power, 80 slaney mel bins, `log(x + 2^-24)`, per-feature |
| normalization over the valid frames. The filterbank and window are the checkpoint's own buffers. Dither is |
| training-only and is not applied. |
| |
| ## How it works |
| |
| ### Why one window |
| |
| In a Core AI bundle the activation arena is sized per compiled shape and allocated for every function in the bundle. |
| A ladder of input-length buckets (the usual way to handle variable-length audio) therefore costs the sum of all its |
| shapes: 768 MB to over 1.1 GB measured for seven buckets. This port compiles exactly one shape, a 512-frame (5.12 s) |
| window, and handles longer audio on the host. |
| |
| ### Why a naive window is wrong, and the fix |
| |
| TitaNet has three reductions over the whole time axis: |
| |
| 1. **five SqueezeExcite gates**: each block scales its channels by `sigmoid(W2 relu(W1 mean_t(x)))` over the entire |
| utterance; |
| 2. **the pooling mean / std** of the encoder output; |
| 3. **the attention softmax over time** in attentive statistics pooling. |
|
|
| Computing these per window gives the wrong function: chunk-local SqueezeExcite alone drops the cosine to about 0.36. |
| The graph here makes all three reductions *injectable* (taken as inputs) and *reportable* (partial sums returned as |
| outputs), so the host can rebuild them exactly: |
|
|
| | Pass | Calls | Rebuilds | |
| |---|---|---| |
| | 1–5 | `scan` per window, gate sums fed back as `s0..s4` | the SqueezeExcite fixed point: block *j*'s mean depends only on gates < *j*, so sweep *k* makes gate *k−1* exact and five sweeps make all five exact | |
| | 6 | `scan` per window | pooling mean / std from running sums `x_sum`, `x_sqsum` | |
| | 7 | `scan` per window, with the pooling stats | the attention softmax, merged across windows by running maximum (`lmax`, `den`, `ax`, `axx`) | |
| | — | `head` once | the 192-d embedding from the attentive mean and std | |
|
|
| Every convolution is stride 1 with "same" padding and the stack's receptive field is 93 frames, so each window carries |
| 46 frames of real context on each side and keeps only its interior; the interiors tile the clip exactly once. A clip |
| of *n* mel frames (10 ms each) uses 1 window if *n* ≤ 512, otherwise ⌈(*n* − 512) / 420⌉ + 1, and |
| **7 × windows + 1** model calls. Host accumulation is float64, and host memory is O(channels), never O(frames). |
|
|
| All five sweeps are needed: the gate residual falls to 6e-4 after two sweeps and *looks* converged while the |
| embedding cosine is still 0.64. Cosine by number of sweeps: 0.066, 0.639, 0.993, 0.99996, 0.9999987. |
|
|
| ### Precision |
|
|
| `wfp16`: weights stored as fp16 (half the bytes), all compute in fp32. A full-fp16 graph was rejected for long |
| audio: its whole-utterance reductions sum thousands of frames in fp16, and the worst-case cosine fell to 0.72 on |
| clips up to 65 s, although it looked fine (0.99998) on 4-second clips. |
|
|
| ### Interface |
|
|
| `scan`: one window of the clip. |
|
|
| | Input | Shape | Meaning | |
| |---|---|---| |
| | `mel` | [1, 80, 512] | the window's normalized log-mel, zero past the clip's end | |
| | `valid` | [1, 1, 512] | 1 for frames inside the clip | |
| | `keep` | [1, 1, 512] | 1 for the window's interior frames, whose statistics this call reports | |
| | `n_inv` | [1, 1] | 1 / (valid frames in the whole clip) | |
| | `s0`..`s3`, `s4` | [1, 1024] ×4, [1, 3072] | SqueezeExcite sums from the previous sweep (zeros on the first) | |
| | `mean`, `std` | [1, 3072] each | pooling statistics for the attention pass (zeros and ones before they are known) | |
|
|
| Outputs, all [1, C]: `se0`..`se4` (this window's SqueezeExcite contributions), `x_sum`, `x_sqsum`, `lmax`, `den`, |
| `ax`, `axx` (C = 3072). |
|
|
| `head`: `mu`, `sg` [1, 3072] → `emb` [1, 192]. |
|
|
| ## License and attribution |
|
|
| This is a derivative of [nvidia/speakerverification_en_titanet_large](https://huggingface.co/nvidia/speakerverification_en_titanet_large) |
| by NVIDIA, released under [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/); this port is released under the |
| same license. |
| Modifications: converted to Core AI with fp16 weight storage, and the network re-authored so its global reductions |
| can be supplied and reported per window (the attention-pooling concat is folded into its first convolution). The |
| weights are NVIDIA's; no retraining or fine-tuning was done. |
|
|
| ```bibtex |
| @inproceedings{koluguri2022titanet, |
| title = {TitaNet: Neural Model for Speaker Representation with 1D Depth-wise Separable Convolutions and Global Context}, |
| author = {Koluguri, Nithin Rao and Park, Taejin and Ginsburg, Boris}, |
| booktitle = {ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)}, |
| pages = {8102--8106}, |
| year = {2022} |
| } |
| ``` |
|
|
