Wataru commited on
Commit
aeb5764
·
verified ·
1 Parent(s): 074b3c0

Model card: tables in the format of the DialogueSidon paper (arXiv)

Browse files
Files changed (1) hide show
  1. README.md +43 -42
README.md CHANGED
@@ -72,48 +72,49 @@ The programs were exported with `torch.export` from PyTorch 2.8.0; load them wit
72
 
73
  ## Evaluation
74
 
75
- The protocol and metrics of the DialogueSidon paper: WER with Whisper large-v3 (CER on CallFriend), NISQA, DNSMOS,
76
- speaker similarity (WavLM), VAD accuracy, and the mean absolute errors of the overlap rate, the gap at turn
77
- changes and the backchannel rate against the ground truth. Both models were scored in the same run on the same
78
- items: Switchboard (100 dialogues of 20 s), OpenDialog (100 recordings under 30 s) and CallFriend (20 dialogues of
79
- 20 s per language). DialogueSidon is the released model (30 sampling steps). On CallFriend, the output tracks are
80
- assigned to the speakers by character errors. **Bold**: the better of the two models.
81
-
82
- **Switchboard**
83
-
84
- | Method | WER (%) ↓ | NISQA ↑ | DNSMOS ↑ | Spk. Sim. ↑ | VAD Acc. ↑ | Ov. MAE ↓ | Gap MAE ↓ | BC MAE ↓ |
85
- |---|---|---|---|---|---|---|---|---|
86
- | Ground Truth | 60.812 | 2.315 | 3.005 | 1.000 | 0.765 | 35.350 | 1.489 | 3.120 |
87
- | DialogueSidon | 14.432 | 3.453 | 3.641 | 0.887 | 0.938 | 2.123 | 0.834 | 2.610 |
88
- | DialogueSidon2 | **12.706** | **3.634** | **3.756** | **0.895** | **0.942** | **1.766** | **0.754** | **2.130** |
89
-
90
- **OpenDialog**
91
-
92
- | Method | WER (%) ↓ | NISQA ↑ | DNSMOS ↑ | Spk. Sim. ↑ | VAD Acc. ↑ | Ov. MAE ↓ | Gap MAE ↓ | BC MAE ↓ |
93
- |---|---|---|---|---|---|---|---|---|
94
- | Ground Truth | 100.286 | 3.826 | 3.615 | 1.000 | 1.000 | 0.000 | 0.000 | 0.000 |
95
- | DialogueSidon | 14.073 | 3.568 | 3.598 | **0.858** | **0.590** | **79.506** | **2.756** | **2.894** |
96
- | DialogueSidon2 | **9.955** | **3.819** | **3.713** | 0.847 | 0.575 | 83.032 | 2.934 | 2.978 |
97
-
98
- **CallFriend**
99
-
100
- | Language | Method | CER (%) ↓ | NISQA ↑ | DNSMOS ↑ | Spk. Sim. ↑ | VAD Acc. ↑ | Ov. MAE ↓ | Gap MAE ↓ | BC MAE ↓ |
101
- |---|---|---|---|---|---|---|---|---|---|
102
- | German | Ground Truth | 0.000 | 2.131 | 3.001 | 1.000 | 1.000 | 0.000 | 0.000 | 0.000 |
103
- | | DialogueSidon | 14.225 | 3.203 | 3.541 | **0.902** | 0.954 | 4.016 | 0.888 | 2.400 |
104
- | | DialogueSidon2 | **12.860** | **3.602** | **3.675** | 0.895 | **0.962** | **3.132** | **0.346** | **1.950** |
105
- | French (Quebec) | Ground Truth | 0.000 | 2.353 | 3.178 | 1.000 | 1.000 | 0.000 | 0.000 | 0.000 |
106
- | | DialogueSidon | **22.683** | 3.393 | 3.639 | 0.906 | **0.948** | **2.352** | 0.753 | **1.650** |
107
- | | DialogueSidon2 | 23.895 | **3.654** | **3.786** | **0.907** | 0.941 | 2.550 | **0.692** | 1.800 |
108
- | Japanese | Ground Truth | 0.000 | 2.177 | 2.897 | 1.000 | 1.000 | 0.000 | 0.000 | 0.000 |
109
- | | DialogueSidon | **19.911** | 3.157 | 3.335 | 0.929 | 0.934 | 4.567 | **0.783** | **1.950** |
110
- | | DialogueSidon2 | 21.411 | **3.530** | **3.486** | **0.941** | **0.940** | **3.212** | 0.962 | 2.250 |
111
- | Spanish | Ground Truth | 0.000 | 2.235 | 2.919 | 1.000 | 1.000 | 0.000 | 0.000 | 0.000 |
112
- | | DialogueSidon | **18.354** | 3.155 | 3.324 | 0.865 | 0.933 | 6.210 | 0.603 | 1.800 |
113
- | | DialogueSidon2 | 19.375 | **3.361** | **3.533** | **0.868** | **0.941** | **4.926** | **0.556** | **1.200** |
114
- | Mandarin | Ground Truth | 0.000 | 2.380 | 3.214 | 1.000 | 1.000 | 0.000 | 0.000 | 0.000 |
115
- | | DialogueSidon | 30.565 | 3.432 | 3.532 | 0.925 | 0.916 | 8.527 | **0.721** | **1.800** |
116
- | | DialogueSidon2 | **25.471** | **3.803** | **3.805** | **0.942** | **0.921** | **7.630** | 1.032 | **1.800** |
 
117
 
118
  ## Training data
119
 
 
72
 
73
  ## Evaluation
74
 
75
+ The evaluation sets and metrics of the [DialogueSidon paper](https://arxiv.org/abs/2604.09344): WER with Whisper
76
+ large-v3 (p-CER on CallFriend), NISQA, DNSMOS, speaker similarity (WavLM) and VAD accuracy. Both models were
77
+ scored in the same run on the same items. DialogueSidon is the released model (30 sampling steps). On CallFriend,
78
+ the output tracks are assigned to the speakers by character errors.
79
+
80
+ **Switchboard (SWB) evaluation set, 100 dialogues of 20 s.** †Noisy does not perform speech separation. Bold: the better of DialogueSidon and DialogueSidon2.
81
+
82
+ | Method | WER (%) ↓ | NISQA ↑ | DNSMOS ↑ | Spk. Sim. ↑ | VAD Acc. ↑ |
83
+ |---|---|---|---|---|---|
84
+ | | *Content* | *Recording Quality* | *Recording Quality* | *Speaker* | *Separation* |
85
+ | Noisy† | 60.812 | 2.315 | 3.005 | — | — |
86
+ | DialogueSidon | 14.432 | 3.453 | 3.641 | 0.887 | 0.938 |
87
+ | DialogueSidon2 | **12.706** | **3.634** | **3.756** | **0.895** | **0.942** |
88
+
89
+ **CallFriend, 20 dialogues of 20 s per language.** †Noisy does not perform speech separation. Bold: the better of DialogueSidon and DialogueSidon2.
90
+
91
+ | Language | Method | p-CER (%) ↓ | NISQA ↑ | DNSMOS ↑ | Spk. Sim. ↑ | VAD Acc. ↑ |
92
+ |---|---|---|---|---|---|---|
93
+ | | | *Content* | *Recording Quality* | *Recording Quality* | *Speaker* | *Separation* |
94
+ | German | Noisy† | — | 2.131 | 3.001 | — | — |
95
+ | | DialogueSidon | 14.225 | 3.203 | 3.541 | **0.902** | 0.954 |
96
+ | | DialogueSidon2 | **12.860** | **3.602** | **3.675** | 0.895 | **0.962** |
97
+ | French | Noisy† | — | 2.353 | 3.178 | — | — |
98
+ | | DialogueSidon | **22.683** | 3.393 | 3.639 | 0.906 | **0.948** |
99
+ | | DialogueSidon2 | 23.895 | **3.654** | **3.786** | **0.907** | 0.941 |
100
+ | Japanese | Noisy† | — | 2.177 | 2.897 | — | — |
101
+ | | DialogueSidon | **19.911** | 3.157 | 3.335 | 0.929 | 0.934 |
102
+ | | DialogueSidon2 | 21.411 | **3.530** | **3.486** | **0.941** | **0.940** |
103
+ | Spanish | Noisy† | — | 2.235 | 2.919 | — | — |
104
+ | | DialogueSidon | **18.354** | 3.155 | 3.324 | 0.865 | 0.933 |
105
+ | | DialogueSidon2 | 19.375 | **3.361** | **3.533** | **0.868** | **0.941** |
106
+ | Mandarin | Noisy† | — | 2.380 | 3.214 | — | — |
107
+ | | DialogueSidon | 30.565 | 3.432 | 3.532 | 0.925 | 0.916 |
108
+ | | DialogueSidon2 | **25.471** | **3.803** | **3.805** | **0.942** | **0.921** |
109
+
110
+ **OpenDialog (in-the-wild data), 100 recordings under 30 s.** †Noisy does not perform speech separation; its metrics are computed on the monaural mixture. Bold: the better of DialogueSidon and DialogueSidon2.
111
+
112
+ | Method | WER (%) ↓ | NISQA ↑ | DNSMOS ↑ |
113
+ |---|---|---|---|
114
+ | | *Content* | *Recording Quality* | *Recording Quality* |
115
+ | Noisy† | — | 3.826 | 3.615 |
116
+ | DialogueSidon | 14.073 | 3.568 | 3.598 |
117
+ | DialogueSidon2 | **9.955** | **3.819** | **3.713** |
118
 
119
  ## Training data
120