huseinzolkepliscicom commited on
Commit
be853d0
·
verified ·
1 Parent(s): 2d43db8

Card: a shorter How we measured (534 to 238 words); numbers and figures unchanged

Browse files
Files changed (1) hide show
  1. README.md +15 -28
README.md CHANGED
@@ -49,8 +49,7 @@ clean speech.
49
  | Resemble Enhance | 44.1 kHz | 4.8 | 8.6 | 48.8 | 19.5 | 15.7 | 2.28 | 3.05 |
50
  | input (telephone audio) | 8 to 16 kHz | 51.3 | 49.2 | 48.8 | | | | |
51
 
52
- Bold marks the best system in each column. Silence filled has no best: it is the counter-metric to speech muted, not a score
53
- (see How we measured). The input row is for reference: unprocessed calls read 48.8 % nCER.
54
 
55
  ![Median spectrum on 58 real calls](assets/spectrum_calls.png)
56
 
@@ -62,32 +61,20 @@ Bold marks the best system in each column. Silence filled has no best: it is the
62
 
63
  ### How we measured
64
 
65
- - **Test audio.** 1,289 in-domain call-centre clips (English, Malay and Mandarin, 9.4 hours) and 58 real calls with human
66
- transcripts (116 legs, one party per channel). Both are customer data and stay private. We publish aggregates only.
67
- - **Same input for everyone.** Every system restores the same files. VoiceFixer and Resemble Enhance output 44.1 kHz; we resample
68
- them to 48 kHz before scoring.
69
- - **Settings.** CallEnhancer v1 and v2: `infer_callcentre.py` defaults (4-second windows, native 48 kHz output). sidon-v0.1: the
70
- released TorchScript model. VoiceFixer: `restore`, mode 0. Resemble Enhance: `enhance` with its command-line defaults
71
- (64 function evaluations, midpoint solver, denoise strength 1.0, temperature 0.5).
72
- - **HBD.** The RMS gap from 4 to 20 kHz between a system's median spectrum and clean speech, both relative to the 1 to 4 kHz band.
73
- - **nCER.** Character error rate of [mesolitica/Malaysian-whisper-large-v3-turbo-v3](https://huggingface.co/mesolitica/Malaysian-whisper-large-v3-turbo-v3)
74
- against the human transcripts, after LLM normalisation, mean of 3 repeats. The two parties are restored separately and mixed at
75
- matched levels. All systems were scored in one session.
76
- - **Speech muted.** The share of speech (20 ms frames) that the output drops by more than 6 dB, on the clips whose reference is
77
- speech. Each signal is measured against its own 90th-percentile frame, so a gain change does not count. A plain band-pass
78
- that deletes nothing scores 5.9 %.
79
- - **Silence filled.** The counter-metric to speech muted. It is the share of frames that are silent in the input (40 dB or more
80
- below its 90th-percentile frame) that the output raises by more than 6 dB. It is not a score. High means energy added to the
81
- pauses. Low is no merit either: an output that pushes its quiet parts down fills little silence and mutes more speech. A change
82
- in loudness balance alone moves the two in opposite directions. In our fault-injection test, compressing an output's dynamic
83
- range by 10 % lowered muting by 1.6 points and raised silence filled by 18; a real deletion raises muting with silence filled
84
- flat. So compare muting at similar silence filled. The shaded band in the deletions figure is where v1 would sit if only its
85
- loudness balance changed.
86
- - **SOTA-MOS.** [Scicom-intl/HighRateMOS-VoiceMOS2025](https://huggingface.co/Scicom-intl/HighRateMOS-VoiceMOS2025) on 300 paired
87
- clips at 48 kHz.
88
- - **One-channel mixes.** The two sides of each of the 58 real calls averaged into one channel at their recorded levels (the sides' speech levels differ by 11.4 dB at the median) and restored as one file. A party's frame counts when its side is within 25 dB of that side's 90th-percentile frame and 15 dB or more above the other side. Muted as for speech muted, with the outputs resampled to 8 kHz so the input and the output share the band.
89
- - **DNSMOS** P.835 overall, the median over the clips it can score. It cannot score the 6 clips under 1 second, or an output whose middle 10 seconds are silent. The medians cover 1,283 clips (sidon-v0.1 1,282, Resemble Enhance 1,243).
90
- - **Silent outputs.** Clips whose whole output stays under -60 dBFS: CallEnhancer v2 0, CallEnhancer v1 0, sidon-v0.1 0, VoiceFixer 0, Resemble Enhance 41. 7 of the 1,289 inputs are silent. Where the input has speech, the speech-muted column counts a silent output as a deletion.
91
 
92
  ## Quick start
93
 
 
49
  | Resemble Enhance | 44.1 kHz | 4.8 | 8.6 | 48.8 | 19.5 | 15.7 | 2.28 | 3.05 |
50
  | input (telephone audio) | 8 to 16 kHz | 51.3 | 49.2 | 48.8 | | | | |
51
 
52
+ Bold marks the best system in each column. Silence filled has no best (see How we measured).
 
53
 
54
  ![Median spectrum on 58 real calls](assets/spectrum_calls.png)
55
 
 
61
 
62
  ### How we measured
63
 
64
+ - **Test audio.** 1,289 call-centre clips (English, Malay and Mandarin, 9.4 hours) and 58 real calls with human transcripts, one
65
+ party per channel. Both are customer data; we publish aggregates only.
66
+ - **Systems.** Each restores the same files with its default settings. 44.1 kHz outputs are resampled to 48 kHz.
67
+ - **nCER.** [Malaysian Whisper](https://huggingface.co/mesolitica/Malaysian-whisper-large-v3-turbo-v3) against the human transcripts,
68
+ after LLM normalisation, mean of 3 runs. Each party is restored on its own channel.
69
+ - **Speech muted.** The share of 20 ms speech frames that the output drops by more than 6 dB. A band-pass that deletes nothing
70
+ scores 5.9 %.
71
+ - **Silence filled.** The share of silent input frames that the output raises by more than 6 dB. A counter-metric, not a score: a
72
+ change in loudness balance trades one against the other, so compare muting at similar silence filled. The shaded band in the
73
+ deletions figure is v1 with only its balance changed.
74
+ - **One channel.** Both sides of each real call mixed into one file at their recorded levels (11.4 dB apart at the median). Speech muted is read on the quieter party, at 8 kHz.
75
+ - **SOTA-MOS** ([HighRateMOS](https://huggingface.co/Scicom-intl/HighRateMOS-VoiceMOS2025)) on 300 clips at 48 kHz.
76
+ - **DNSMOS** P.835 overall, the median over the clips it can score (1,243 to 1,283 of 1,289).
77
+ - **Silent outputs.** Resemble Enhance returns silence (under -60 dBFS) on 41 clips; the other systems never do. Speech muted counts these as deletions.
 
 
 
 
 
 
 
 
 
 
 
 
78
 
79
  ## Quick start
80