Automatic Speech Recognition
NeMo
English
speech
multi-talker-asr
target-speaker-asr
speaker-diarization
meeting-transcription
long-form
parakeet
fastconformer
ctc
Instructions to use BUT-FIT/Pandemonium with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- NeMo
How to use BUT-FIT/Pandemonium with NeMo:
import nemo.collections.asr as nemo_asr asr_model = nemo_asr.models.ASRModel.from_pretrained("BUT-FIT/Pandemonium") transcriptions = asr_model.transcribe(["file.wav"]) - Notebooks
- Google Colab
- Kaggle
|
Download README.md from BUT-FIT/Pandemonium: direct link, hf CLI and curl.
- Browser
- Download file 20.6 kB
-
https://huggingface.co/BUT-FIT/Pandemonium/resolve/main/README.md
- Command line
-
hf download hf://BUT-FIT/Pandemonium/README.md
-
curl -L -o README.md https://huggingface.co/BUT-FIT/Pandemonium/resolve/main/README.md
20.6 kB
| license: cc-by-4.0 | |
| language: | |
| - en | |
| datasets: | |
| - openslr/librispeech_asr | |
| - espnet/yodas2 | |
| - microsoft/NOTSOFAR | |
| - edinburghcstr/ami | |
| pipeline_tag: automatic-speech-recognition | |
| library_name: nemo | |
| base_model: nvidia/parakeet-tdt-0.6b-v3 | |
| tags: | |
| - speech | |
| - automatic-speech-recognition | |
| - multi-talker-asr | |
| - target-speaker-asr | |
| - speaker-diarization | |
| - meeting-transcription | |
| - long-form | |
| - parakeet | |
| - fastconformer | |
| - ctc | |
| # Pandemonium | |
| **Who said what, in a whole meeting, in one pass.** Pandemonium is a long-form, diarization-conditioned | |
| multi-talker ASR model from [BUT Speech@FIT](https://github.com/BUTSpeechFIT). Given a recording and a | |
| diarization of it, it returns a time-stamped transcript for every speaker, decoding all speakers of the | |
| conversation together so that each speaker's transcript is informed by the others. | |
| It is described in **[Pandemonium: Longform Target-Speaker Speech Recognition with Cross-Speaker | |
| Interactions](https://huggingface.co/BUT-FIT/Pandemonium/blob/main/Pandemonium_ICASSP2027.pdf)** (Polok et al., | |
| submitted to ICASSP 2027; [PDF](https://huggingface.co/BUT-FIT/Pandemonium/blob/main/Pandemonium_ICASSP2027.pdf)). | |
| Please refer to the paper for the method, the training recipe and the full evaluation, and | |
| [cite it](#citation) if you use the model. Code: [BUTSpeechFIT/Pandemonium](https://github.com/BUTSpeechFIT/Pandemonium). | |
| Also released with the paper: the backbone without cross-speaker interaction, [DiCoP v2](https://huggingface.co/BUT-FIT/DiCoP-v2) | |
| (TDT and CTC variants), and the recipe for the YODAS2 pseudo-labels, [YODAS-2-MT](https://huggingface.co/datasets/BUT-FIT/YODAS-2-MT). | |
| > *Pandemonium*: the collective noun for a flock of parakeets — or chaos. | |
| | | | | |
| |---|---| | |
| | Task | speaker-attributed (multi-talker) ASR, English | | |
| | Input | mono 16 kHz audio + per-speaker activity from any diarizer (soft probabilities or hard segments) | | |
| | Output | one time-stamped word sequence per speaker | | |
| | Backbone | [Parakeet-TDT-0.6b-v3](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3) FastConformer encoder (24 layers, d=1024), local attention [256, 256] + 4 global tokens | | |
| | Head | frozen Parakeet joiner used as a frame-synchronous CTC head, greedy decoding | | |
| | Parameters | ≈0.7 B | | |
| | Speakers | up to 8 per recording | | |
| | Context | whole recordings in one pass (evaluated on meetings of up to 1 h) | | |
| | Speed | RTFx 426× including Nemotron-3-Diarization, 690× ASR alone (released pipeline, [below](#speed)) | | |
| ## How it works | |
| Pandemonium builds on **diarization-conditioned target-speaker ASR** (DiCoW, | |
| [SE-DiCoW](https://huggingface.co/BUT-FIT/SE-DiCoW)). For each speaker found by a diarizer, the model is | |
| conditioned on a frame-level **STNO mask** — the probabilities that a frame is *silence*, *target only*, | |
| *non-target only*, or *overlap* — which modulates every encoder layer through class-specific affine | |
| transforms (FDDT). One recognizer is thereby turned into a recognizer of *that* speaker, while remaining | |
| aware of the others. | |
| On top of that, Pandemonium adds three things: | |
| 1. **Long-form backbone (DiCoP).** The conditioning is moved from Whisper's fixed 30 s windows into | |
| Parakeet's FastConformer, whose local attention processes entire meetings on a single GPU. Because the | |
| TDT transducer loss does not scale to multi-minute targets, the model is trained with CTC through the | |
| (frozen) TDT joiner with its predictor contribution zeroed — no new parameters. | |
| 2. **Cross-speaker attention (CSA).** A conversation's speakers are encoded as one batch, and after | |
| encoder layers 8, 12, 16 and 20 each frame attends *across the speaker axis* (multi-head attention, | |
| no positional encoding, so the number of speakers is not fixed). Speakers are no longer decoded in | |
| isolation: overlaps, turn-taking and backchannels can be resolved jointly. | |
| 3. **STNO mask refiner.** After layers 12, 16 and 20 the model re-estimates each speaker's activity from | |
| the cross-speaker-informed states, rebuilds the STNO mask from it, and blends it into the conditioning | |
| of the deeper layers through a learned gate. This lets it repair diarization errors, and separate | |
| speakers that the input mask labels identically in overlapped frames. | |
| Both new blocks start as near-identities, so training warm-starts cleanly from Parakeet. | |
| ## Training | |
| Three stages, each starting from the previous one: | |
| | stage | data | hours | | |
| |---|---|---| | |
| | 1. synthetic | [FastMSS](https://arxiv.org/abs/2605.15442) mixtures of up to 8 LibriSpeech speakers (2k h) + Fisher conversations restored with Sidon and force-aligned (2k h) | 4.0k | | |
| | 2. pseudo-labeled | multi-speaker YODAS2 conversations in the 23 languages Parakeet-v3 supports, automatically diarized and transcribed by DiCoW v3.3 conditioned on that diarization ([recipe and statistics](https://huggingface.co/datasets/BUT-FIT/YODAS-2-MT)) | 4.6k | | |
| | 3. real | NOTSOFAR-1 and AMI single-distant-microphone training sets | 115 | | |
| Training conversations are capped at 8 speakers. The diarization mask is randomly corrupted during | |
| training so that the refiner has errors to correct; the refiner is trained at a 10× learning rate. | |
| This checkpoint is the best epoch by validation cpWER (epoch 16, cpWER 0.1597). The exact training | |
| configuration is included as `pandemonium.yaml` (recipe) and `model_config.yaml` (the configuration | |
| stored in the checkpoint). | |
| ### Training data and licences | |
| | source | used for | licence | | |
| |---|---|---| | |
| | [Parakeet-TDT-0.6b-v3](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3) | initialization (encoder, joiner, tokenizer) | CC-BY-4.0 | | |
| | [LibriSpeech](https://www.openslr.org/12) | stage 1, synthetic mixtures | CC-BY-4.0 | | |
| | Fisher English ([LDC2004S13](https://catalog.ldc.upenn.edu/LDC2004S13), [LDC2005S13](https://catalog.ldc.upenn.edu/LDC2005S13)) | stage 1, restored with [Sidon](https://github.com/sarulab-speech/Sidon) (MIT, preprocessing only) | LDC user agreement | | |
| | [YODAS2](https://huggingface.co/datasets/espnet/yodas2) | stage 2, pseudo-labeled ([recipe](https://huggingface.co/datasets/BUT-FIT/YODAS-2-MT); labels not redistributed) | CC-BY-3.0 | | |
| | [NOTSOFAR-1](https://github.com/microsoft/NOTSOFAR1-Challenge) | stage 3 | CC-BY-4.0 | | |
| | [AMI](https://groups.inf.ed.ac.uk/ami/corpus/) | stage 3 | CC-BY-4.0 | | |
| No training audio or transcripts are redistributed with the model. | |
| ## Results | |
| All numbers are cpWER or tcpWER in %, lower is better, with greedy decoding, full recordings, and | |
| whisper_nsf text normalization. NSF = NOTSOFAR-1, Mx6 = Mixer 6, SBC = Santa Barbara Corpus (conversation | |
| and task subsets with 2–8 speakers); M1 and M2 are two further meeting corpora, not named here for licensing | |
| reasons. NOTSOFAR-1 and AMI are in-domain; the rest are out-of-domain. Commercial APIs marked "API A–D" are | |
| anonymized. | |
| ### Paper (cpWER) | |
| | system | NSF | AMI | M1 | Mx6 | SBC | M2 | Avg | | |
| |---|---|---|---|---|---|---|---| | |
| | *Oracle diarization* | | | | | | | | | |
| | DiCoW v3.3 | 15.5 | 14.5 | 14.1 | 11.4 | 15.0 | 27.1 | 16.3 | | |
| | SE-DiCoW | 15.3 | 14.4 | 14.3 | 11.7 | 15.2 | 26.2 | 16.2 | | |
| | DiCoP (no CSA) | 17.3 | 14.5 | 14.4 | 11.9 | 16.0 | 26.1 | 16.7 | | |
| | **Pandemonium** | **14.1** | **13.4** | **13.3** | **11.1** | **13.8** | **22.4** | **14.7** | | |
| | + MOVER (all distant mics) | 10.8 | 12.6 | 11.9 | 9.7 | 13.8 | 19.4 | 13.0 | | |
| | *Predicted diarization (pyannoteAI Precision-3)* | | | | | | | | | |
| | DiCoW v3.3 | 23.0 | 17.3 | 18.6 | 11.5 | 19.3 | 30.0 | 20.0 | | |
| | SE-DiCoW | 22.8 | 17.3 | 18.2 | 11.8 | 19.6 | 29.3 | 19.8 | | |
| | DiCoP (no CSA) | 23.2 | 16.9 | 18.0 | 12.1 | 19.4 | 28.9 | 19.8 | | |
| | **Pandemonium** | **22.4** | **16.8** | 18.1 | 11.7 | **18.6** | **27.4** | **19.2** | | |
| | + MOVER (all distant mics) | 19.0 | 14.5 | 15.7 | 10.2 | 18.6 | 24.8 | 17.1 | | |
| | best published single-mic (Sep 2026) | 22.8 | 17.6 | 20.9 | 14.5 | – | 34.0 | – | | |
| With predicted diarization, Pandemonium is below the best published single-microphone result on every | |
| set where one exists. | |
| ### With Nemotron-3-Diarization (the released pipeline) | |
| Conditioning on the *soft* speaker probabilities of | |
| [Nemotron-3-Diarization](https://huggingface.co/nvidia/Nemotron-3-Diarization) (offline setting, no | |
| thresholding), on 251 full recordings, 54.5 h: | |
| | | NSF | AMI | M1 | Mx6 | SBC | M2 | Avg | | |
| |---|---|---|---|---|---|---|---| | |
| | cpWER | 17.93 | 16.14 | 14.53 | 12.03 | 18.67 | 23.52 | 17.14 | | |
| | tcpWER (5 s collar) | 18.23 | 16.76 | 14.82 | 12.28 | 19.09 | 24.90 | 17.68 | | |
| Soft conditioning beats thresholded segments from the same diarizer (NOTSOFAR-1: 17.93 vs 18.52 cpWER), | |
| even though the model was trained on hard masks only. | |
| ### Comparison with other systems (October 2026) | |
| Same 251 recordings, references, normalization and scorer as the table above; tcpWER, 5 s collar. | |
| Every system uses its own diarization unless stated. | |
| | system | NSF | AMI | M1 | Mx6 | SBC | M2 | **Avg** | RTFx | | |
| |---|---|---|---|---|---|---|---|---| | |
| | **Pandemonium** + Nemotron-3-Diarization | **18.23** | **16.76** | 14.82 | 12.28 | **19.09** | **24.90** | **17.68** | **426×** | | |
| | **Pandemonium** + pyannoteAI Precision-3 | 23.41 | 17.85 | 18.90 | 11.93 | 19.14 | 29.89 | 20.19 | 95× API ∥ | | |
| | SE-DiCoW + DiariZen | 25.83 | 19.28 | 20.73 | 12.20 | 21.58 | 35.98 | 22.60 | 28.9× | | |
| | Nemotron-3-Diarization + MT-Parakeet § | 31.56 | 20.38 | **12.93** § | 18.56 | 26.83 | 35.65 | 24.32 | ~60× | | |
| | MOSS-Transcribe-Diarize | 28.10 | 32.14 | 25.60 | 11.15 | 22.48 | 39.90 | 26.56 | 6.1× | | |
| | API A | 34.11 | 28.65 | 31.31 | 12.36 | 24.31 | 29.67 | 26.74 | 1.0× API ¶ | | |
| | API B | 34.84 | 27.41 | 32.73 | 14.24 | 24.05 | 31.61 | 27.48 | 67× API | | |
| | Xiaomi CocktailASR + DiariZen | 32.78 | 23.19 | 28.33 | 16.18 | 26.90 | 38.45 | 27.64 | 8.2× | | |
| | MAI-Transcribe-2 ‡ | 32.97 | 38.06 | 39.32 | 10.91 | 28.44 | 28.59 | 29.72 | 214× API | | |
| | ElevenLabs Scribe v2 | 37.41 | 35.54 | 43.76 | **9.63** | 24.50 | 36.43 | 31.21 | 81× API | | |
| | API C | 42.53 | 34.40 | 34.56 | 16.16 | 32.10 | 42.16 | 33.65 | 141× API | | |
| | VibeVoice-ASR | 36.33 | 41.88 | 42.48 | 12.01 | 29.53 | 64.73 | 37.83 | 6.6× | | |
| | Gemini 3.5 Transcribe †| 47.75 | 45.33 | 52.03 | 21.27 | 41.37 | 48.46 | 42.70 | 49× API | | |
| | API D | 52.86 | 39.37 | 31.19 | 25.42 | 37.88 | 77.24 | 43.99 | 224× API | | |
| | Mistral Voxtral ‡ | 58.04 | 42.95 | 46.04 | 19.56 | 39.03 | 63.62 | 44.87 | 112× API | | |
| | OpenAI gpt-4o-transcribe-diarize | 56.42 | 60.07 | 59.99 | 28.11 | 45.76 | 62.66 | 52.17 | 2.7× API | | |
| - †Gemini is scored with **cpWER**, as its word timestamps drift on long audio. It is run on windows of | |
| up to 15 min with its own speaker labels, linked across windows through DiariZen. | |
| - ‡ Returns only segment-level timestamps, which costs it under tcpWER. | |
| - § Its training data may include the M1 evaluation sessions. | |
| - ¶ Streaming API, which accepts audio no faster than real time. | |
| - ∥ Word error rates use the Precision-3 diarization of the paper, conditioned as hard segments, decoded | |
| as full recordings (cpWER 19.14 average, matching the paper's 19.2 above). RTFx is the public pyannoteAI API | |
| (92.7 s for upload and diarization, client wall-clock) plus Pandemonium on one GPU (12.6 s); the public | |
| API's output differs slightly from the paper's diarization (11.5% DER between the two). | |
| - RTFx is audio duration divided by processing time, *including diarization*, on one median-length | |
| recording per set (6 recordings, 10,022 s). Local models are timed on one GPU with warm-up excluded; | |
| API figures are client wall-clock times including upload, and are not directly comparable. | |
| - SBC uses 44 of its 60 sessions. | |
| ### Headset mixes | |
| The individual headset (IHM) channels of a meeting summed into one signal: overlapped speech without | |
| far-field reverberation. AMI uses the official `Mix-Headset` recordings; NOTSOFAR-1 has none, so its | |
| close-talk channels are summed. Same references and scorer as above; tcpWER / cpWER, 5 s collar. | |
| | system | NSF-IHM-mix | AMI-IHM-mix | **Avg** | | |
| |---|---|---|---| | |
| | *Pandemonium + oracle activity* | *8.84 / 8.81* | *9.71 / 9.59* | *9.28 / 9.20* | | |
| | **Pandemonium** + Nemotron-3-Diarization | **12.33 / 12.14** | **11.46 / 11.16** | **11.90 / 11.65** | | |
| | SE-DiCoW + DiariZen | 17.68 / 16.90 | 15.12 / 14.42 | 16.40 / 15.66 | | |
| | DiCoW v3.3 + DiariZen | 17.83 / 17.05 | 15.20 / 14.48 | 16.52 / 15.77 | | |
| | Nemotron-3-Diarization + MT-Parakeet | 20.20 / 19.71 | 13.83 / 13.37 | 17.02 / 16.54 | | |
| | Xiaomi CocktailASR + DiariZen | 23.09 / 22.07 | 15.89 / 15.17 | 19.49 / 18.62 | | |
| | MOSS-Transcribe-Diarize | 23.90 / 23.63 | 20.08 / 19.72 | 21.99 / 21.68 | | |
| | VibeVoice-ASR | 26.38 / 25.94 | 23.71 / 23.16 | 25.05 / 24.55 | | |
| ### Short fully-overlapped mixtures | |
| Read speech mixed at full overlap, 10–20 s per mixture: LibriSpeechMix (test-clean, 1–3 speakers) and | |
| LibriMix (test-clean, 2–3 speakers, clean and with WHAM! noise). cpWER; tcpWER is nearly identical at | |
| these lengths. | |
| | system | LSMix 1 | LSMix 2 | LSMix 3 | L2Mix | L2Mix noisy | L3Mix | L3Mix noisy | **Avg** | | |
| |---|---|---|---|---|---|---|---|---| | |
| | SE-DiCoW + DiariZen | **1.82** | **3.13** | 21.94 | **3.49** | **9.88** | **26.61** | **35.09** | **14.57** | | |
| | **Pandemonium** + Nemotron-3-Diarization | 2.34 | 3.20 | **5.17** | 6.76 | 17.81 | 28.02 | 41.76 | 15.01 | | |
| | DiCoW v3.3 + DiariZen | 1.89 | 3.65 | 22.63 | 4.04 | 10.50 | 28.45 | 36.27 | 15.35 | | |
| | *Pandemonium + oracle activity* | *2.27* | *3.37* | *5.48* | *10.19* | *23.25* | *33.73* | *46.62* | *17.84* | | |
| | Xiaomi CocktailASR + DiariZen | 1.93 | 5.41 | 19.57 | 13.53 | 29.36 | 36.16 | 50.93 | 22.41 | | |
| | MOSS-Transcribe-Diarize | 3.59 | 11.29 | 19.06 | 24.82 | 36.19 | 55.89 | 64.52 | 30.77 | | |
| | Nemotron-3-Diarization + MT-Parakeet | 2.63 | 7.43 | 13.01 | 24.68 | 43.79 | 62.19 | 73.31 | 32.43 | | |
| | VibeVoice-ASR | 2.18 | 50.90 | 73.79 | 57.87 | 64.81 | 82.06 | 87.83 | 59.92 | | |
| Oracle activity is rasterized from the reference segments, which on LibriMix are looser than a diarizer's | |
| output; this is the likely reason it trails the Nemotron-conditioned run there. | |
| ### Single-speaker speech | |
| Multi-talker training costs accuracy on ordinary single-speaker speech. Open ASR Leaderboard test sets | |
| (TED-LIUM is no longer distributed with them), WER in %, the leaderboard's Whisper English normalizer. | |
| Parakeet-TDT-0.6b-v3 is the base model, run by us (matching its leaderboard scores). | |
| | system | LS clean | LS other | VoxPopuli | Earnings-22 | AMI-IHM | GigaSpeech | SPGISpeech | **Avg** | | |
| |---|---|---|---|---|---|---|---|---| | |
| | Parakeet-TDT-0.6b-v3 | **1.94** | **3.59** | **6.10** | **11.18** | **11.34** | **9.53** | **3.98** | **6.81** | | |
| | Pandemonium, whole recording as one speaker | 2.30 | 5.07 | 15.57 | 16.00 | 13.76 | 10.26 | 5.59 | 9.79 | | |
| | Pandemonium + Nemotron-3-Diarization | 2.35 | 5.00 | 15.26 | 16.07 | 14.13 | 10.89 | 5.54 | 9.89 | | |
| Most of the gap is on VoxPopuli: greedy CTC decoding through the frozen TDT joiner drops word-internal | |
| subword pieces ("suffering" → "suering", "crisis" → "cr"). Pandemonium also outputs lowercase text without punctuation. | |
| For single-speaker audio, Parakeet-TDT-0.6b-v3 remains the better choice. | |
| ## Speed | |
| - **Paper**: RTFx 330 for the ASR alone, against 405 for the same backbone without the interaction | |
| modules and 4–8× slower DiCoW/SE-DiCoW (Fig. 2 of the paper). Measured with the research decoding code, | |
| for comparing architectures; the released inference code is faster. | |
| - **Released pipeline**: RTFx 426 including Nemotron-3-Diarization; on the 6 timing recordings above, | |
| 9.0 s of diarization and 14.5 s of transcription for 167 min of audio (690× for the transcription | |
| alone). The whole NOTSOFAR-1 eval set (16.7 h) takes 117 s to diarize and about 3 min to transcribe. | |
| - **With pyannoteAI Precision-3** (API): RTFx 95 on the same recordings, 92.7 s in the API and 12.6 s of | |
| transcription. | |
| - **Memory**: a 60-minute, 8-speaker recording peaks at about 12 GiB of GPU memory in bf16. | |
| ## Usage | |
| The model needs a diarization of the recording. The reference pipeline uses | |
| [Nemotron-3-Diarization](https://huggingface.co/nvidia/Nemotron-3-Diarization) and conditions on its soft | |
| speaker probabilities. The inference code is a small self-contained package | |
| (`pandemonium/` + `pipeline.py`, in `demo/` of the [code repository](https://github.com/BUTSpeechFIT/Pandemonium)). | |
| ```bash | |
| conda create -n pandemonium python=3.12 -y && conda activate pandemonium | |
| pip install Cython packaging | |
| pip install -r requirements.txt # pins NeMo to a main-branch commit; Nemotron-3-Diarization needs it | |
| ``` | |
| ```python | |
| import librosa | |
| from pipeline import DemoPipeline, utterances | |
| pipe = DemoPipeline() # downloads Pandemonium + Nemotron-3-Diarization | |
| audio, _ = librosa.load("meeting.wav", sr=16000, mono=True) | |
| diar = pipe.diarize(audio) # per-speaker probabilities, up to 8 speakers | |
| words = pipe.transcribe(audio, diar) # soft conditioning; soft=False uses thresholded segments | |
| for u in utterances(words): # words grouped at pauses > 1 s, interleaved by time | |
| print(f"[{u['start']:7.1f}-{u['end']:7.1f}] speaker {u['speaker']}: {u['text']}") | |
| ``` | |
| To use a different diarizer, pass its per-speaker activity at the model's 80 ms frame rate directly: | |
| ```python | |
| from pandemonium import Pandemonium | |
| model = Pandemonium.from_pretrained() # bf16 on CUDA by default | |
| T = Pandemonium.num_frames(len(audio)) | |
| activity = ... # (num_speakers, T) array in [0, 1]: probabilities or 0/1 | |
| words = model.transcribe(audio, activity) # list (per speaker) of Word(text, start, end) | |
| ``` | |
| ## Files | |
| | file | contents | | |
| |---|---| | |
| | `cp_wer=0.1597-epoch=16.ckpt` | weights (PyTorch Lightning checkpoint: `state_dict` + `hyper_parameters`, fp32, 2.8 GB) | | |
| | `model_config.yaml` | the configuration stored in the checkpoint | | |
| | `pandemonium.yaml` | training recipe | | |
| | `tokenizer.model` | SentencePiece tokenizer, shared with Parakeet-TDT-0.6b-v3 | | |
| | `Pandemonium_ICASSP2027.pdf` | the paper | | |
| When loading the checkpoint in the training code rather than the inference package, set | |
| `model.num_speaker_slots=8 +model.encoder.subsampling_conv_chunking_factor=2`. | |
| ## Limitations | |
| - **English only.** The backbone is multilingual, but the real and evaluation data are English; other | |
| languages are untested. Mandarin is not supported by the Parakeet tokenizer. | |
| - **Depends on the diarization.** Missed speakers are not transcribed, and merged speakers are | |
| transcribed as one. Pandemonium's refiner corrects mask errors but cannot add speakers. | |
| - **At most 8 speakers** per recording. | |
| - **Single channel.** Multi-microphone recordings are best decoded per microphone and combined with | |
| MOVER (see the paper). | |
| - **Trained on hard masks.** Soft diarizer probabilities work better in practice (see above), but they are | |
| outside the training distribution. | |
| - Greedy decoding only; verbatim style (fillers and backchannels are transcribed), which may differ from | |
| references that drop them. | |
| ## Citation | |
| If you use Pandemonium, please cite the [paper](https://huggingface.co/BUT-FIT/Pandemonium/blob/main/Pandemonium_ICASSP2027.pdf): | |
| ```bibtex | |
| @inproceedings{polok2027pandemonium, | |
| title = {Pandemonium: Longform Target-Speaker Speech Recognition with Cross-Speaker Interactions}, | |
| author = {Polok, Alexander and Klement, Dominik and Medennikov, Ivan and Laurent, Antoine and | |
| P{\'a}lka, Petr and Cornell, Samuele and Watanabe, Shinji and Khudanpur, Sanjeev and | |
| Balam, Jagadeesh and {\v{C}}ernock{\'y}, Jan and Burget, Luk{\'a}{\v{s}} and Wiesner, Matthew}, | |
| booktitle = {Submitted to ICASSP}, | |
| year = {2027} | |
| } | |
| ``` | |
| ## Acknowledgements | |
| This work was supported by JSALT 2026 at Johns Hopkins University (NSF CCRI Grant No. 2120435, Google | |
| DeepMind, JHU HLTCOE, JHU AI2AI, ACL), by the Ministry of Education, Youth and Sports of the Czech Republic | |
| through the OP JAK project "Linguistics, Artificial Intelligence and Language and Speech Technologies: from | |
| Research to Applications" (CZ.02.01.01/00/23_020/0008518), and by the Brno Ph.D. Talent Scholarship | |
| Programme. Computational resources were provided by e-INFRA CZ (ID: 90254) and by IDRIS under GENCI | |
| allocations AD011012527, A0201012177 and AD011016176. | |
| ## License | |
| [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/). Pandemonium is a fine-tuned derivative of | |
| [Parakeet-TDT-0.6b-v3](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3) (CC-BY-4.0, NVIDIA); see | |
| [Training data and licences](#training-data-and-licences) for the data it was trained on. | |
| ## Contact | |
| [ipoloka@fit.vut.cz](mailto:ipoloka@fit.vut.cz) · [BUT Speech@FIT](https://github.com/BUTSpeechFIT), Brno | |
| University of Technology | |