Model card: tables in the format of the DialogueSidon paper (arXiv)
Browse files
README.md
CHANGED
|
@@ -72,48 +72,49 @@ The programs were exported with `torch.export` from PyTorch 2.8.0; load them wit
|
|
| 72 |
|
| 73 |
## Evaluation
|
| 74 |
|
| 75 |
-
The
|
| 76 |
-
|
| 77 |
-
|
| 78 |
-
|
| 79 |
-
|
| 80 |
-
|
| 81 |
-
|
| 82 |
-
|
| 83 |
-
|
| 84 |
-
|
|
| 85 |
-
|
|
| 86 |
-
|
|
| 87 |
-
|
|
| 88 |
-
|
| 89 |
-
|
| 90 |
-
|
| 91 |
-
|
| 92 |
-
|
|
| 93 |
-
|
|
| 94 |
-
|
|
| 95 |
-
| DialogueSidon | 14.
|
| 96 |
-
| DialogueSidon2 | **
|
| 97 |
-
|
| 98 |
-
**
|
| 99 |
-
|
| 100 |
-
|
|
| 101 |
-
|
|
| 102 |
-
|
|
| 103 |
-
|
|
| 104 |
-
| |
|
| 105 |
-
|
|
| 106 |
-
|
|
| 107 |
-
| |
|
| 108 |
-
|
|
| 109 |
-
|
| 110 |
-
|
| 111 |
-
|
| 112 |
-
|
|
| 113 |
-
|
|
| 114 |
-
|
|
| 115 |
-
|
|
| 116 |
-
|
|
|
|
|
| 117 |
|
| 118 |
## Training data
|
| 119 |
|
|
|
|
| 72 |
|
| 73 |
## Evaluation
|
| 74 |
|
| 75 |
+
The evaluation sets and metrics of the [DialogueSidon paper](https://arxiv.org/abs/2604.09344): WER with Whisper
|
| 76 |
+
large-v3 (p-CER on CallFriend), NISQA, DNSMOS, speaker similarity (WavLM) and VAD accuracy. Both models were
|
| 77 |
+
scored in the same run on the same items. DialogueSidon is the released model (30 sampling steps). On CallFriend,
|
| 78 |
+
the output tracks are assigned to the speakers by character errors.
|
| 79 |
+
|
| 80 |
+
**Switchboard (SWB) evaluation set, 100 dialogues of 20 s.** †Noisy does not perform speech separation. Bold: the better of DialogueSidon and DialogueSidon2.
|
| 81 |
+
|
| 82 |
+
| Method | WER (%) ↓ | NISQA ↑ | DNSMOS ↑ | Spk. Sim. ↑ | VAD Acc. ↑ |
|
| 83 |
+
|---|---|---|---|---|---|
|
| 84 |
+
| | *Content* | *Recording Quality* | *Recording Quality* | *Speaker* | *Separation* |
|
| 85 |
+
| Noisy† | 60.812 | 2.315 | 3.005 | — | — |
|
| 86 |
+
| DialogueSidon | 14.432 | 3.453 | 3.641 | 0.887 | 0.938 |
|
| 87 |
+
| DialogueSidon2 | **12.706** | **3.634** | **3.756** | **0.895** | **0.942** |
|
| 88 |
+
|
| 89 |
+
**CallFriend, 20 dialogues of 20 s per language.** †Noisy does not perform speech separation. Bold: the better of DialogueSidon and DialogueSidon2.
|
| 90 |
+
|
| 91 |
+
| Language | Method | p-CER (%) ↓ | NISQA ↑ | DNSMOS ↑ | Spk. Sim. ↑ | VAD Acc. ↑ |
|
| 92 |
+
|---|---|---|---|---|---|---|
|
| 93 |
+
| | | *Content* | *Recording Quality* | *Recording Quality* | *Speaker* | *Separation* |
|
| 94 |
+
| German | Noisy† | — | 2.131 | 3.001 | — | — |
|
| 95 |
+
| | DialogueSidon | 14.225 | 3.203 | 3.541 | **0.902** | 0.954 |
|
| 96 |
+
| | DialogueSidon2 | **12.860** | **3.602** | **3.675** | 0.895 | **0.962** |
|
| 97 |
+
| French | Noisy† | — | 2.353 | 3.178 | — | — |
|
| 98 |
+
| | DialogueSidon | **22.683** | 3.393 | 3.639 | 0.906 | **0.948** |
|
| 99 |
+
| | DialogueSidon2 | 23.895 | **3.654** | **3.786** | **0.907** | 0.941 |
|
| 100 |
+
| Japanese | Noisy† | — | 2.177 | 2.897 | — | — |
|
| 101 |
+
| | DialogueSidon | **19.911** | 3.157 | 3.335 | 0.929 | 0.934 |
|
| 102 |
+
| | DialogueSidon2 | 21.411 | **3.530** | **3.486** | **0.941** | **0.940** |
|
| 103 |
+
| Spanish | Noisy† | — | 2.235 | 2.919 | — | — |
|
| 104 |
+
| | DialogueSidon | **18.354** | 3.155 | 3.324 | 0.865 | 0.933 |
|
| 105 |
+
| | DialogueSidon2 | 19.375 | **3.361** | **3.533** | **0.868** | **0.941** |
|
| 106 |
+
| Mandarin | Noisy† | — | 2.380 | 3.214 | — | — |
|
| 107 |
+
| | DialogueSidon | 30.565 | 3.432 | 3.532 | 0.925 | 0.916 |
|
| 108 |
+
| | DialogueSidon2 | **25.471** | **3.803** | **3.805** | **0.942** | **0.921** |
|
| 109 |
+
|
| 110 |
+
**OpenDialog (in-the-wild data), 100 recordings under 30 s.** †Noisy does not perform speech separation; its metrics are computed on the monaural mixture. Bold: the better of DialogueSidon and DialogueSidon2.
|
| 111 |
+
|
| 112 |
+
| Method | WER (%) ↓ | NISQA ↑ | DNSMOS ↑ |
|
| 113 |
+
|---|---|---|---|
|
| 114 |
+
| | *Content* | *Recording Quality* | *Recording Quality* |
|
| 115 |
+
| Noisy† | — | 3.826 | 3.615 |
|
| 116 |
+
| DialogueSidon | 14.073 | 3.568 | 3.598 |
|
| 117 |
+
| DialogueSidon2 | **9.955** | **3.819** | **3.713** |
|
| 118 |
|
| 119 |
## Training data
|
| 120 |
|