Pandemonium / README.md
Lakoc's picture
Card: the repository is not gated
9290959 verified
|
Raw History Blame Contribute Delete
20.6 kB
---
license: cc-by-4.0
language:
- en
datasets:
- openslr/librispeech_asr
- espnet/yodas2
- microsoft/NOTSOFAR
- edinburghcstr/ami
pipeline_tag: automatic-speech-recognition
library_name: nemo
base_model: nvidia/parakeet-tdt-0.6b-v3
tags:
- speech
- automatic-speech-recognition
- multi-talker-asr
- target-speaker-asr
- speaker-diarization
- meeting-transcription
- long-form
- parakeet
- fastconformer
- ctc
---
# Pandemonium
**Who said what, in a whole meeting, in one pass.** Pandemonium is a long-form, diarization-conditioned
multi-talker ASR model from [BUT Speech@FIT](https://github.com/BUTSpeechFIT). Given a recording and a
diarization of it, it returns a time-stamped transcript for every speaker, decoding all speakers of the
conversation together so that each speaker's transcript is informed by the others.
It is described in **[Pandemonium: Longform Target-Speaker Speech Recognition with Cross-Speaker
Interactions](https://huggingface.co/BUT-FIT/Pandemonium/blob/main/Pandemonium_ICASSP2027.pdf)** (Polok et al.,
submitted to ICASSP 2027; [PDF](https://huggingface.co/BUT-FIT/Pandemonium/blob/main/Pandemonium_ICASSP2027.pdf)).
Please refer to the paper for the method, the training recipe and the full evaluation, and
[cite it](#citation) if you use the model. Code: [BUTSpeechFIT/Pandemonium](https://github.com/BUTSpeechFIT/Pandemonium).
Also released with the paper: the backbone without cross-speaker interaction, [DiCoP v2](https://huggingface.co/BUT-FIT/DiCoP-v2)
(TDT and CTC variants), and the recipe for the YODAS2 pseudo-labels, [YODAS-2-MT](https://huggingface.co/datasets/BUT-FIT/YODAS-2-MT).
> *Pandemonium*: the collective noun for a flock of parakeets — or chaos.
| | |
|---|---|
| Task | speaker-attributed (multi-talker) ASR, English |
| Input | mono 16 kHz audio + per-speaker activity from any diarizer (soft probabilities or hard segments) |
| Output | one time-stamped word sequence per speaker |
| Backbone | [Parakeet-TDT-0.6b-v3](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3) FastConformer encoder (24 layers, d=1024), local attention [256, 256] + 4 global tokens |
| Head | frozen Parakeet joiner used as a frame-synchronous CTC head, greedy decoding |
| Parameters | ≈0.7 B |
| Speakers | up to 8 per recording |
| Context | whole recordings in one pass (evaluated on meetings of up to 1 h) |
| Speed | RTFx 426× including Nemotron-3-Diarization, 690× ASR alone (released pipeline, [below](#speed)) |
## How it works
Pandemonium builds on **diarization-conditioned target-speaker ASR** (DiCoW,
[SE-DiCoW](https://huggingface.co/BUT-FIT/SE-DiCoW)). For each speaker found by a diarizer, the model is
conditioned on a frame-level **STNO mask** — the probabilities that a frame is *silence*, *target only*,
*non-target only*, or *overlap* — which modulates every encoder layer through class-specific affine
transforms (FDDT). One recognizer is thereby turned into a recognizer of *that* speaker, while remaining
aware of the others.
On top of that, Pandemonium adds three things:
1. **Long-form backbone (DiCoP).** The conditioning is moved from Whisper's fixed 30 s windows into
Parakeet's FastConformer, whose local attention processes entire meetings on a single GPU. Because the
TDT transducer loss does not scale to multi-minute targets, the model is trained with CTC through the
(frozen) TDT joiner with its predictor contribution zeroed — no new parameters.
2. **Cross-speaker attention (CSA).** A conversation's speakers are encoded as one batch, and after
encoder layers 8, 12, 16 and 20 each frame attends *across the speaker axis* (multi-head attention,
no positional encoding, so the number of speakers is not fixed). Speakers are no longer decoded in
isolation: overlaps, turn-taking and backchannels can be resolved jointly.
3. **STNO mask refiner.** After layers 12, 16 and 20 the model re-estimates each speaker's activity from
the cross-speaker-informed states, rebuilds the STNO mask from it, and blends it into the conditioning
of the deeper layers through a learned gate. This lets it repair diarization errors, and separate
speakers that the input mask labels identically in overlapped frames.
Both new blocks start as near-identities, so training warm-starts cleanly from Parakeet.
## Training
Three stages, each starting from the previous one:
| stage | data | hours |
|---|---|---|
| 1. synthetic | [FastMSS](https://arxiv.org/abs/2605.15442) mixtures of up to 8 LibriSpeech speakers (2k h) + Fisher conversations restored with Sidon and force-aligned (2k h) | 4.0k |
| 2. pseudo-labeled | multi-speaker YODAS2 conversations in the 23 languages Parakeet-v3 supports, automatically diarized and transcribed by DiCoW v3.3 conditioned on that diarization ([recipe and statistics](https://huggingface.co/datasets/BUT-FIT/YODAS-2-MT)) | 4.6k |
| 3. real | NOTSOFAR-1 and AMI single-distant-microphone training sets | 115 |
Training conversations are capped at 8 speakers. The diarization mask is randomly corrupted during
training so that the refiner has errors to correct; the refiner is trained at a 10× learning rate.
This checkpoint is the best epoch by validation cpWER (epoch 16, cpWER 0.1597). The exact training
configuration is included as `pandemonium.yaml` (recipe) and `model_config.yaml` (the configuration
stored in the checkpoint).
### Training data and licences
| source | used for | licence |
|---|---|---|
| [Parakeet-TDT-0.6b-v3](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3) | initialization (encoder, joiner, tokenizer) | CC-BY-4.0 |
| [LibriSpeech](https://www.openslr.org/12) | stage 1, synthetic mixtures | CC-BY-4.0 |
| Fisher English ([LDC2004S13](https://catalog.ldc.upenn.edu/LDC2004S13), [LDC2005S13](https://catalog.ldc.upenn.edu/LDC2005S13)) | stage 1, restored with [Sidon](https://github.com/sarulab-speech/Sidon) (MIT, preprocessing only) | LDC user agreement |
| [YODAS2](https://huggingface.co/datasets/espnet/yodas2) | stage 2, pseudo-labeled ([recipe](https://huggingface.co/datasets/BUT-FIT/YODAS-2-MT); labels not redistributed) | CC-BY-3.0 |
| [NOTSOFAR-1](https://github.com/microsoft/NOTSOFAR1-Challenge) | stage 3 | CC-BY-4.0 |
| [AMI](https://groups.inf.ed.ac.uk/ami/corpus/) | stage 3 | CC-BY-4.0 |
No training audio or transcripts are redistributed with the model.
## Results
All numbers are cpWER or tcpWER in %, lower is better, with greedy decoding, full recordings, and
whisper_nsf text normalization. NSF = NOTSOFAR-1, Mx6 = Mixer 6, SBC = Santa Barbara Corpus (conversation
and task subsets with 2–8 speakers); M1 and M2 are two further meeting corpora, not named here for licensing
reasons. NOTSOFAR-1 and AMI are in-domain; the rest are out-of-domain. Commercial APIs marked "API A–D" are
anonymized.
### Paper (cpWER)
| system | NSF | AMI | M1 | Mx6 | SBC | M2 | Avg |
|---|---|---|---|---|---|---|---|
| *Oracle diarization* | | | | | | | |
| DiCoW v3.3 | 15.5 | 14.5 | 14.1 | 11.4 | 15.0 | 27.1 | 16.3 |
| SE-DiCoW | 15.3 | 14.4 | 14.3 | 11.7 | 15.2 | 26.2 | 16.2 |
| DiCoP (no CSA) | 17.3 | 14.5 | 14.4 | 11.9 | 16.0 | 26.1 | 16.7 |
| **Pandemonium** | **14.1** | **13.4** | **13.3** | **11.1** | **13.8** | **22.4** | **14.7** |
| + MOVER (all distant mics) | 10.8 | 12.6 | 11.9 | 9.7 | 13.8 | 19.4 | 13.0 |
| *Predicted diarization (pyannoteAI Precision-3)* | | | | | | | |
| DiCoW v3.3 | 23.0 | 17.3 | 18.6 | 11.5 | 19.3 | 30.0 | 20.0 |
| SE-DiCoW | 22.8 | 17.3 | 18.2 | 11.8 | 19.6 | 29.3 | 19.8 |
| DiCoP (no CSA) | 23.2 | 16.9 | 18.0 | 12.1 | 19.4 | 28.9 | 19.8 |
| **Pandemonium** | **22.4** | **16.8** | 18.1 | 11.7 | **18.6** | **27.4** | **19.2** |
| + MOVER (all distant mics) | 19.0 | 14.5 | 15.7 | 10.2 | 18.6 | 24.8 | 17.1 |
| best published single-mic (Sep 2026) | 22.8 | 17.6 | 20.9 | 14.5 | – | 34.0 | – |
With predicted diarization, Pandemonium is below the best published single-microphone result on every
set where one exists.
### With Nemotron-3-Diarization (the released pipeline)
Conditioning on the *soft* speaker probabilities of
[Nemotron-3-Diarization](https://huggingface.co/nvidia/Nemotron-3-Diarization) (offline setting, no
thresholding), on 251 full recordings, 54.5 h:
| | NSF | AMI | M1 | Mx6 | SBC | M2 | Avg |
|---|---|---|---|---|---|---|---|
| cpWER | 17.93 | 16.14 | 14.53 | 12.03 | 18.67 | 23.52 | 17.14 |
| tcpWER (5 s collar) | 18.23 | 16.76 | 14.82 | 12.28 | 19.09 | 24.90 | 17.68 |
Soft conditioning beats thresholded segments from the same diarizer (NOTSOFAR-1: 17.93 vs 18.52 cpWER),
even though the model was trained on hard masks only.
### Comparison with other systems (October 2026)
Same 251 recordings, references, normalization and scorer as the table above; tcpWER, 5 s collar.
Every system uses its own diarization unless stated.
| system | NSF | AMI | M1 | Mx6 | SBC | M2 | **Avg** | RTFx |
|---|---|---|---|---|---|---|---|---|
| **Pandemonium** + Nemotron-3-Diarization | **18.23** | **16.76** | 14.82 | 12.28 | **19.09** | **24.90** | **17.68** | **426×** |
| **Pandemonium** + pyannoteAI Precision-3 | 23.41 | 17.85 | 18.90 | 11.93 | 19.14 | 29.89 | 20.19 | 95× API ∥ |
| SE-DiCoW + DiariZen | 25.83 | 19.28 | 20.73 | 12.20 | 21.58 | 35.98 | 22.60 | 28.9× |
| Nemotron-3-Diarization + MT-Parakeet § | 31.56 | 20.38 | **12.93** § | 18.56 | 26.83 | 35.65 | 24.32 | ~60× |
| MOSS-Transcribe-Diarize | 28.10 | 32.14 | 25.60 | 11.15 | 22.48 | 39.90 | 26.56 | 6.1× |
| API A | 34.11 | 28.65 | 31.31 | 12.36 | 24.31 | 29.67 | 26.74 | 1.0× API ¶ |
| API B | 34.84 | 27.41 | 32.73 | 14.24 | 24.05 | 31.61 | 27.48 | 67× API |
| Xiaomi CocktailASR + DiariZen | 32.78 | 23.19 | 28.33 | 16.18 | 26.90 | 38.45 | 27.64 | 8.2× |
| MAI-Transcribe-2 ‡ | 32.97 | 38.06 | 39.32 | 10.91 | 28.44 | 28.59 | 29.72 | 214× API |
| ElevenLabs Scribe v2 | 37.41 | 35.54 | 43.76 | **9.63** | 24.50 | 36.43 | 31.21 | 81× API |
| API C | 42.53 | 34.40 | 34.56 | 16.16 | 32.10 | 42.16 | 33.65 | 141× API |
| VibeVoice-ASR | 36.33 | 41.88 | 42.48 | 12.01 | 29.53 | 64.73 | 37.83 | 6.6× |
| Gemini 3.5 Transcribe † | 47.75 | 45.33 | 52.03 | 21.27 | 41.37 | 48.46 | 42.70 | 49× API |
| API D | 52.86 | 39.37 | 31.19 | 25.42 | 37.88 | 77.24 | 43.99 | 224× API |
| Mistral Voxtral ‡ | 58.04 | 42.95 | 46.04 | 19.56 | 39.03 | 63.62 | 44.87 | 112× API |
| OpenAI gpt-4o-transcribe-diarize | 56.42 | 60.07 | 59.99 | 28.11 | 45.76 | 62.66 | 52.17 | 2.7× API |
- † Gemini is scored with **cpWER**, as its word timestamps drift on long audio. It is run on windows of
up to 15 min with its own speaker labels, linked across windows through DiariZen.
- ‡ Returns only segment-level timestamps, which costs it under tcpWER.
- § Its training data may include the M1 evaluation sessions.
- ¶ Streaming API, which accepts audio no faster than real time.
- ∥ Word error rates use the Precision-3 diarization of the paper, conditioned as hard segments, decoded
as full recordings (cpWER 19.14 average, matching the paper's 19.2 above). RTFx is the public pyannoteAI API
(92.7 s for upload and diarization, client wall-clock) plus Pandemonium on one GPU (12.6 s); the public
API's output differs slightly from the paper's diarization (11.5% DER between the two).
- RTFx is audio duration divided by processing time, *including diarization*, on one median-length
recording per set (6 recordings, 10,022 s). Local models are timed on one GPU with warm-up excluded;
API figures are client wall-clock times including upload, and are not directly comparable.
- SBC uses 44 of its 60 sessions.
### Headset mixes
The individual headset (IHM) channels of a meeting summed into one signal: overlapped speech without
far-field reverberation. AMI uses the official `Mix-Headset` recordings; NOTSOFAR-1 has none, so its
close-talk channels are summed. Same references and scorer as above; tcpWER / cpWER, 5 s collar.
| system | NSF-IHM-mix | AMI-IHM-mix | **Avg** |
|---|---|---|---|
| *Pandemonium + oracle activity* | *8.84 / 8.81* | *9.71 / 9.59* | *9.28 / 9.20* |
| **Pandemonium** + Nemotron-3-Diarization | **12.33 / 12.14** | **11.46 / 11.16** | **11.90 / 11.65** |
| SE-DiCoW + DiariZen | 17.68 / 16.90 | 15.12 / 14.42 | 16.40 / 15.66 |
| DiCoW v3.3 + DiariZen | 17.83 / 17.05 | 15.20 / 14.48 | 16.52 / 15.77 |
| Nemotron-3-Diarization + MT-Parakeet | 20.20 / 19.71 | 13.83 / 13.37 | 17.02 / 16.54 |
| Xiaomi CocktailASR + DiariZen | 23.09 / 22.07 | 15.89 / 15.17 | 19.49 / 18.62 |
| MOSS-Transcribe-Diarize | 23.90 / 23.63 | 20.08 / 19.72 | 21.99 / 21.68 |
| VibeVoice-ASR | 26.38 / 25.94 | 23.71 / 23.16 | 25.05 / 24.55 |
### Short fully-overlapped mixtures
Read speech mixed at full overlap, 10–20 s per mixture: LibriSpeechMix (test-clean, 1–3 speakers) and
LibriMix (test-clean, 2–3 speakers, clean and with WHAM! noise). cpWER; tcpWER is nearly identical at
these lengths.
| system | LSMix 1 | LSMix 2 | LSMix 3 | L2Mix | L2Mix noisy | L3Mix | L3Mix noisy | **Avg** |
|---|---|---|---|---|---|---|---|---|
| SE-DiCoW + DiariZen | **1.82** | **3.13** | 21.94 | **3.49** | **9.88** | **26.61** | **35.09** | **14.57** |
| **Pandemonium** + Nemotron-3-Diarization | 2.34 | 3.20 | **5.17** | 6.76 | 17.81 | 28.02 | 41.76 | 15.01 |
| DiCoW v3.3 + DiariZen | 1.89 | 3.65 | 22.63 | 4.04 | 10.50 | 28.45 | 36.27 | 15.35 |
| *Pandemonium + oracle activity* | *2.27* | *3.37* | *5.48* | *10.19* | *23.25* | *33.73* | *46.62* | *17.84* |
| Xiaomi CocktailASR + DiariZen | 1.93 | 5.41 | 19.57 | 13.53 | 29.36 | 36.16 | 50.93 | 22.41 |
| MOSS-Transcribe-Diarize | 3.59 | 11.29 | 19.06 | 24.82 | 36.19 | 55.89 | 64.52 | 30.77 |
| Nemotron-3-Diarization + MT-Parakeet | 2.63 | 7.43 | 13.01 | 24.68 | 43.79 | 62.19 | 73.31 | 32.43 |
| VibeVoice-ASR | 2.18 | 50.90 | 73.79 | 57.87 | 64.81 | 82.06 | 87.83 | 59.92 |
Oracle activity is rasterized from the reference segments, which on LibriMix are looser than a diarizer's
output; this is the likely reason it trails the Nemotron-conditioned run there.
### Single-speaker speech
Multi-talker training costs accuracy on ordinary single-speaker speech. Open ASR Leaderboard test sets
(TED-LIUM is no longer distributed with them), WER in %, the leaderboard's Whisper English normalizer.
Parakeet-TDT-0.6b-v3 is the base model, run by us (matching its leaderboard scores).
| system | LS clean | LS other | VoxPopuli | Earnings-22 | AMI-IHM | GigaSpeech | SPGISpeech | **Avg** |
|---|---|---|---|---|---|---|---|---|
| Parakeet-TDT-0.6b-v3 | **1.94** | **3.59** | **6.10** | **11.18** | **11.34** | **9.53** | **3.98** | **6.81** |
| Pandemonium, whole recording as one speaker | 2.30 | 5.07 | 15.57 | 16.00 | 13.76 | 10.26 | 5.59 | 9.79 |
| Pandemonium + Nemotron-3-Diarization | 2.35 | 5.00 | 15.26 | 16.07 | 14.13 | 10.89 | 5.54 | 9.89 |
Most of the gap is on VoxPopuli: greedy CTC decoding through the frozen TDT joiner drops word-internal
subword pieces ("suffering" → "suering", "crisis" → "cr"). Pandemonium also outputs lowercase text without punctuation.
For single-speaker audio, Parakeet-TDT-0.6b-v3 remains the better choice.
## Speed
- **Paper**: RTFx 330 for the ASR alone, against 405 for the same backbone without the interaction
modules and 4–8× slower DiCoW/SE-DiCoW (Fig. 2 of the paper). Measured with the research decoding code,
for comparing architectures; the released inference code is faster.
- **Released pipeline**: RTFx 426 including Nemotron-3-Diarization; on the 6 timing recordings above,
9.0 s of diarization and 14.5 s of transcription for 167 min of audio (690× for the transcription
alone). The whole NOTSOFAR-1 eval set (16.7 h) takes 117 s to diarize and about 3 min to transcribe.
- **With pyannoteAI Precision-3** (API): RTFx 95 on the same recordings, 92.7 s in the API and 12.6 s of
transcription.
- **Memory**: a 60-minute, 8-speaker recording peaks at about 12 GiB of GPU memory in bf16.
## Usage
The model needs a diarization of the recording. The reference pipeline uses
[Nemotron-3-Diarization](https://huggingface.co/nvidia/Nemotron-3-Diarization) and conditions on its soft
speaker probabilities. The inference code is a small self-contained package
(`pandemonium/` + `pipeline.py`, in `demo/` of the [code repository](https://github.com/BUTSpeechFIT/Pandemonium)).
```bash
conda create -n pandemonium python=3.12 -y && conda activate pandemonium
pip install Cython packaging
pip install -r requirements.txt # pins NeMo to a main-branch commit; Nemotron-3-Diarization needs it
```
```python
import librosa
from pipeline import DemoPipeline, utterances
pipe = DemoPipeline() # downloads Pandemonium + Nemotron-3-Diarization
audio, _ = librosa.load("meeting.wav", sr=16000, mono=True)
diar = pipe.diarize(audio) # per-speaker probabilities, up to 8 speakers
words = pipe.transcribe(audio, diar) # soft conditioning; soft=False uses thresholded segments
for u in utterances(words): # words grouped at pauses > 1 s, interleaved by time
print(f"[{u['start']:7.1f}-{u['end']:7.1f}] speaker {u['speaker']}: {u['text']}")
```
To use a different diarizer, pass its per-speaker activity at the model's 80 ms frame rate directly:
```python
from pandemonium import Pandemonium
model = Pandemonium.from_pretrained() # bf16 on CUDA by default
T = Pandemonium.num_frames(len(audio))
activity = ... # (num_speakers, T) array in [0, 1]: probabilities or 0/1
words = model.transcribe(audio, activity) # list (per speaker) of Word(text, start, end)
```
## Files
| file | contents |
|---|---|
| `cp_wer=0.1597-epoch=16.ckpt` | weights (PyTorch Lightning checkpoint: `state_dict` + `hyper_parameters`, fp32, 2.8 GB) |
| `model_config.yaml` | the configuration stored in the checkpoint |
| `pandemonium.yaml` | training recipe |
| `tokenizer.model` | SentencePiece tokenizer, shared with Parakeet-TDT-0.6b-v3 |
| `Pandemonium_ICASSP2027.pdf` | the paper |
When loading the checkpoint in the training code rather than the inference package, set
`model.num_speaker_slots=8 +model.encoder.subsampling_conv_chunking_factor=2`.
## Limitations
- **English only.** The backbone is multilingual, but the real and evaluation data are English; other
languages are untested. Mandarin is not supported by the Parakeet tokenizer.
- **Depends on the diarization.** Missed speakers are not transcribed, and merged speakers are
transcribed as one. Pandemonium's refiner corrects mask errors but cannot add speakers.
- **At most 8 speakers** per recording.
- **Single channel.** Multi-microphone recordings are best decoded per microphone and combined with
MOVER (see the paper).
- **Trained on hard masks.** Soft diarizer probabilities work better in practice (see above), but they are
outside the training distribution.
- Greedy decoding only; verbatim style (fillers and backchannels are transcribed), which may differ from
references that drop them.
## Citation
If you use Pandemonium, please cite the [paper](https://huggingface.co/BUT-FIT/Pandemonium/blob/main/Pandemonium_ICASSP2027.pdf):
```bibtex
@inproceedings{polok2027pandemonium,
title = {Pandemonium: Longform Target-Speaker Speech Recognition with Cross-Speaker Interactions},
author = {Polok, Alexander and Klement, Dominik and Medennikov, Ivan and Laurent, Antoine and
P{\'a}lka, Petr and Cornell, Samuele and Watanabe, Shinji and Khudanpur, Sanjeev and
Balam, Jagadeesh and {\v{C}}ernock{\'y}, Jan and Burget, Luk{\'a}{\v{s}} and Wiesner, Matthew},
booktitle = {Submitted to ICASSP},
year = {2027}
}
```
## Acknowledgements
This work was supported by JSALT 2026 at Johns Hopkins University (NSF CCRI Grant No. 2120435, Google
DeepMind, JHU HLTCOE, JHU AI2AI, ACL), by the Ministry of Education, Youth and Sports of the Czech Republic
through the OP JAK project "Linguistics, Artificial Intelligence and Language and Speech Technologies: from
Research to Applications" (CZ.02.01.01/00/23_020/0008518), and by the Brno Ph.D. Talent Scholarship
Programme. Computational resources were provided by e-INFRA CZ (ID: 90254) and by IDRIS under GENCI
allocations AD011012527, A0201012177 and AD011016176.
## License
[CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/). Pandemonium is a fine-tuned derivative of
[Parakeet-TDT-0.6b-v3](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3) (CC-BY-4.0, NVIDIA); see
[Training data and licences](#training-data-and-licences) for the data it was trained on.
## Contact
[ipoloka@fit.vut.cz](mailto:ipoloka@fit.vut.cz) · [BUT Speech@FIT](https://github.com/BUTSpeechFIT), Brno
University of Technology