Nemotron 3 Diarization: Advancing the Accuracy-Efficiency Frontier in Streaming Speaker Diarization

Streaming Sortformer introduced an arrival-order speaker cache to maintain speaker identities across chunks in end-to-end streaming speaker diarization, with a single model supporting configurable latency profiles spanning offline operation to ultra-low-latency streaming. Nemotron 3 Diarization addresses the original model's four-speaker limitation by extending support to eight speakers while improving both accuracy and computational efficiency. The previous FastConformer-Transformer stack is streamlined into a pure Transformer encoder. A learned upsampling layer then maps the encoder's coarse 80-ms representations to 10-ms resolution, enabling fine-grained predictions. Training leverages large-scale simulated mixtures and adds activity and phantom-speaker losses to the standard Sortformer objectives. Despite doubling speaker capacity over its predecessor, the new architecture reduces training memory by a factor of 2.8, makes training 2.2times faster, and delivers up to 6.4times higher inference throughput. Nemotron 3 Diarization establishes a new state of the art in DER on DIHARD III and CALLHOME. When used to condition DiCoW, it also achieves state-of-the-art tcpWER on the single-microphone conditions of AMI and NOTSOFAR-1. Training code is open source, and model weights are released under the permissive OpenMDW-1.1 license.