Card: a shorter How we measured (534 to 238 words); numbers and figures unchanged
Browse files
README.md
CHANGED
|
@@ -49,8 +49,7 @@ clean speech.
|
|
| 49 |
| Resemble Enhance | 44.1 kHz | 4.8 | 8.6 | 48.8 | 19.5 | 15.7 | 2.28 | 3.05 |
|
| 50 |
| input (telephone audio) | 8 to 16 kHz | 51.3 | 49.2 | 48.8 | | | | |
|
| 51 |
|
| 52 |
-
Bold marks the best system in each column. Silence filled has no best
|
| 53 |
-
(see How we measured). The input row is for reference: unprocessed calls read 48.8 % nCER.
|
| 54 |
|
| 55 |

|
| 56 |
|
|
@@ -62,32 +61,20 @@ Bold marks the best system in each column. Silence filled has no best: it is the
|
|
| 62 |
|
| 63 |
### How we measured
|
| 64 |
|
| 65 |
-
- **Test audio.** 1,289
|
| 66 |
-
|
| 67 |
-
- **
|
| 68 |
-
|
| 69 |
-
|
| 70 |
-
|
| 71 |
-
|
| 72 |
-
- **
|
| 73 |
-
|
| 74 |
-
|
| 75 |
-
|
| 76 |
-
- **
|
| 77 |
-
|
| 78 |
-
|
| 79 |
-
- **Silence filled.** The counter-metric to speech muted. It is the share of frames that are silent in the input (40 dB or more
|
| 80 |
-
below its 90th-percentile frame) that the output raises by more than 6 dB. It is not a score. High means energy added to the
|
| 81 |
-
pauses. Low is no merit either: an output that pushes its quiet parts down fills little silence and mutes more speech. A change
|
| 82 |
-
in loudness balance alone moves the two in opposite directions. In our fault-injection test, compressing an output's dynamic
|
| 83 |
-
range by 10 % lowered muting by 1.6 points and raised silence filled by 18; a real deletion raises muting with silence filled
|
| 84 |
-
flat. So compare muting at similar silence filled. The shaded band in the deletions figure is where v1 would sit if only its
|
| 85 |
-
loudness balance changed.
|
| 86 |
-
- **SOTA-MOS.** [Scicom-intl/HighRateMOS-VoiceMOS2025](https://huggingface.co/Scicom-intl/HighRateMOS-VoiceMOS2025) on 300 paired
|
| 87 |
-
clips at 48 kHz.
|
| 88 |
-
- **One-channel mixes.** The two sides of each of the 58 real calls averaged into one channel at their recorded levels (the sides' speech levels differ by 11.4 dB at the median) and restored as one file. A party's frame counts when its side is within 25 dB of that side's 90th-percentile frame and 15 dB or more above the other side. Muted as for speech muted, with the outputs resampled to 8 kHz so the input and the output share the band.
|
| 89 |
-
- **DNSMOS** P.835 overall, the median over the clips it can score. It cannot score the 6 clips under 1 second, or an output whose middle 10 seconds are silent. The medians cover 1,283 clips (sidon-v0.1 1,282, Resemble Enhance 1,243).
|
| 90 |
-
- **Silent outputs.** Clips whose whole output stays under -60 dBFS: CallEnhancer v2 0, CallEnhancer v1 0, sidon-v0.1 0, VoiceFixer 0, Resemble Enhance 41. 7 of the 1,289 inputs are silent. Where the input has speech, the speech-muted column counts a silent output as a deletion.
|
| 91 |
|
| 92 |
## Quick start
|
| 93 |
|
|
|
|
| 49 |
| Resemble Enhance | 44.1 kHz | 4.8 | 8.6 | 48.8 | 19.5 | 15.7 | 2.28 | 3.05 |
|
| 50 |
| input (telephone audio) | 8 to 16 kHz | 51.3 | 49.2 | 48.8 | | | | |
|
| 51 |
|
| 52 |
+
Bold marks the best system in each column. Silence filled has no best (see How we measured).
|
|
|
|
| 53 |
|
| 54 |

|
| 55 |
|
|
|
|
| 61 |
|
| 62 |
### How we measured
|
| 63 |
|
| 64 |
+
- **Test audio.** 1,289 call-centre clips (English, Malay and Mandarin, 9.4 hours) and 58 real calls with human transcripts, one
|
| 65 |
+
party per channel. Both are customer data; we publish aggregates only.
|
| 66 |
+
- **Systems.** Each restores the same files with its default settings. 44.1 kHz outputs are resampled to 48 kHz.
|
| 67 |
+
- **nCER.** [Malaysian Whisper](https://huggingface.co/mesolitica/Malaysian-whisper-large-v3-turbo-v3) against the human transcripts,
|
| 68 |
+
after LLM normalisation, mean of 3 runs. Each party is restored on its own channel.
|
| 69 |
+
- **Speech muted.** The share of 20 ms speech frames that the output drops by more than 6 dB. A band-pass that deletes nothing
|
| 70 |
+
scores 5.9 %.
|
| 71 |
+
- **Silence filled.** The share of silent input frames that the output raises by more than 6 dB. A counter-metric, not a score: a
|
| 72 |
+
change in loudness balance trades one against the other, so compare muting at similar silence filled. The shaded band in the
|
| 73 |
+
deletions figure is v1 with only its balance changed.
|
| 74 |
+
- **One channel.** Both sides of each real call mixed into one file at their recorded levels (11.4 dB apart at the median). Speech muted is read on the quieter party, at 8 kHz.
|
| 75 |
+
- **SOTA-MOS** ([HighRateMOS](https://huggingface.co/Scicom-intl/HighRateMOS-VoiceMOS2025)) on 300 clips at 48 kHz.
|
| 76 |
+
- **DNSMOS** P.835 overall, the median over the clips it can score (1,243 to 1,283 of 1,289).
|
| 77 |
+
- **Silent outputs.** Resemble Enhance returns silence (under -60 dBFS) on 41 clips; the other systems never do. Speech muted counts these as deletions.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 78 |
|
| 79 |
## Quick start
|
| 80 |
|