eek/nemotron3-diarization-ro-en

Nemotron-3-Diarization fine-tuned on Romanian and English conversations

A fine-tune of nvidia/Nemotron-3-Diarization (100M parameters, up to 8 speakers) for Romanian and English conversations, including people talking over each other. It answers who speaks when; pair it with the companion speech-to-text model eek/whisper-turbo-ro-en, whose transcribe.py --speakers gives a speaker-labelled transcript in one command.

Research model, fine-tuned on a private dataset that is not distributed.

This is the second version. The first (revision ft-d1) was trained only on machine speaker labels that almost never mark overlapping speech, and it learned to ignore overlap. This version also trains on simulated conversations with exact overlap labels. To load the first version, pass revision="ft-d1" to hf_hub_download.

Quick start

Nemotron-3-Diarization needs NVIDIA NeMo Speech 3.1, which is not released as a package yet; install NeMo from the commit this model was trained and tested with:

pip install torch torchaudio
pip install "nemo_toolkit[asr] @ git+https://github.com/NVIDIA-NeMo/Speech@cf724ac337d1ebc7d0dda1e23fb80916f52927a5"

from huggingface_hub import hf_hub_download
from nemo.collections.asr.models import SortformerEncLabelModel

path = hf_hub_download("eek/nemotron3-diarization-ro-en", "nemotron3-diarization-ro-en.nemo")
model = SortformerEncLabelModel.restore_from(path, map_location="cuda").eval()  # or "cpu"
segments = model.diarize(audio=["meeting.wav"], batch_size=1)[0]  # 16 kHz mono WAV
for line in segments:  # "start end speaker_k", seconds
    print(line)

Convert other formats first, for example ffmpeg -i in.m4a -ac 1 -ar 16000 meeting.wav. transcribe.py in the speech-to-text repository does the conversion, the diarization and the word-to-speaker assignment.

Benchmarks

Diarization error rate (%), 0.25 s collar, overlapping speech scored. Lower is better.

Test	Reference	Nemotron-3 (base)	ft-d1 (previous)	This model
Private Romanian conversations, 3 h, up to 4 speakers	machine labels	14.04	6.19	6.65
Private English conversations, 3 h, up to 4 speakers	machine labels	18.32	11.53	12.42
Speaker count exactly right (ro / en)	machine labels	75.5% / 65.6%	79.9% / 74.3%	79.1% / 72.6%
Three-speaker household conversation, 5 min: DER / speaker confusion	human	17.9 / 0.42	11.9 / 1.05	11.8 / 0.29
Two-speaker conversations in 12 languages, no overlap	exact (simulated)	12.6	7.4	0.6
Same, light overlap (0.3–1 s at turn changes)	exact	12.2	7.5	1.9
Same, heavy overlap (second speaker starts 40–70% into the first)	exact	13.6	25.9	3.0
Heavy overlap: share of overlapping speech detected (ro / en / de)	exact	70% / 71% / 71%	0.1% / 13% / 17%	87% / 86% / 83%

• Simulated conversations. They are built from FLEURS utterances in Romanian, English, German, French, Spanish, Italian, Portuguese, Polish, Hungarian, Russian, Ukrainian and Turkish, with speakers paired by voice-embedding distance, so their labels are exact. The voices, sentences and ten of the languages are unseen in training. But the test conversations are assembled the same way as this model's simulated training conversations, which is why it is nearly perfect on them without overlap. The overlap gains hold in every language.
• Household conversation. The reference is word-level, so turns are joined across pauses of up to 1.5 s. Joined at 0.5 s instead, DER is 10.6 (base) / 18.9 (ft-d1) / 18.0 (this model), because the fine-tunes count short pauses inside a phrase as speech, as their training labels do.
• Real conversations. On them the model predicts little overlap (0.07% of speech on the private Romanian set, whose machine labels mark 0.02%). How well it detects real-world overlap has not yet been measured against human labels.

For reference, Streaming Sortformer v2 (4 speakers) scores 14.09 / 17.81 on the two private sets.

Training

One pass from the base model (11,471 steps, batch 4, lr 1e-5 with a cosine schedule, bf16) over 589 h:

• 350 h of real Romanian (250 h) and English (100 h) conversation sessions of up to 90 s, with machine-generated speaker labels;
• 239 h of simulated conversations (Romanian 171 h, English 68 h), built from single-speaker phrases of the same private data:
  • 2–4 speakers from different recordings, far apart in voice-embedding space;
  • 40% of turn changes overlapping by 0.2–3 s;
  • short backchannels from another speaker inside a quarter of the turns;
  • speaker levels within ±5 dB and a shared simulated room in half of the sessions;
  • about 9% of speech overlapped.

Training on the simulated conversations alone predicts far too much overlap on real audio (31.6% false alarm on the household conversation); the real sessions keep it calibrated. The labels count short pauses inside a phrase as speech, and the model adopted that convention.

Licence

OpenMDW-1.1, the licence of the base model (NVIDIA Nemotron-3-Diarization), for the whole model. Anyone may use, modify and redistribute it without restriction, commercial use included, under copyright, patent and database rights. The one condition is that a redistribution includes a copy of LICENSE. Model outputs carry no obligations. The fine-tuning data is private and is not distributed.