RetentionLabs commited on
Commit
fa72cc2
Β·
verified Β·
1 Parent(s): 40d6c52

docs(model): specify evaluator precision and bounded repeatability

Browse files
Files changed (1) hide show
  1. README.md +20 -0
README.md CHANGED
@@ -146,6 +146,20 @@ The authors' full ESD protocol comprises 1,200 seen-speaker pairs and 300 unseen
146
 
147
  Results and setup: [matched full ESD benchmark on W&B](https://wandb.ai/haan_modeling/vadweaver/runs/6958b357).
148
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
149
  #### Seen speakers, 1,200 pairs
150
 
151
  | Precision | WER Whisper (%) ↓ | WER wav2vec2 (%) ↓ | WER avg (%) ↓ | SECS Resemblyzer ↑ | SECS WavLM ↑ | SECS avg ↑ | ECA (%) ↑ | EECS ↑ | SVAS ↑ |
@@ -183,10 +197,16 @@ Exact utterance overlap between the 240-recording selection and the full benchma
183
 
184
  The BF16 estimator timestep input is part of the matched runtime contract. The FP32 scalar integration schedule is retained as solver metadata, without an FP32 inference-weight copy. No controlled inference-speed comparison is reported.
185
 
 
 
 
 
186
  ### Export verification
187
 
188
  A fresh `python -I` process loaded only `AutoProcessor` and `AutoModel` from this repository at code/weight revision `39f692e1ca0fc42b0d29589c946aa3aaed265f39`, with no project-package import or generation hooks. For ESD seen clip `0014_001095` and unseen clip `0013_001433`, processor tensors, generated mels and postprocessed waveforms exactly matched the saved strict BF16/H64 reference pipeline, including the lazy internal BF16 BigVGAN. Verification seeds 777700 and 777701 used stride-preserving `randn_like` noise; this two-clip parity check uses different noise from the full benchmark and establishes export parity, not quality equivalence. Packed state matched the frozen benchmark before and after generation. No optimizer step was performed.
189
 
 
 
190
  ### Paper context
191
 
192
  The following published EmoSphere++ results are from Table II (seen speakers) and Table III (unseen speakers) of [Cho et al.](https://ieeexplore.ieee.org/document/10965917), using the paper's ESD protocol, vocoder and evaluation setup. They provide source context separately from this conversion's matched BF16/H64 results. Different vocoder weights and evaluation details prevent treating the paper rows as matched comparisons to this conversion.
 
146
 
147
  Results and setup: [matched full ESD benchmark on W&B](https://wandb.ai/haan_modeling/vadweaver/runs/6958b357).
148
 
149
+ #### Metric protocol
150
+
151
+ Synthesis uses BF16 for unquantized parameters, reference encoders, floating model inputs and BigVGAN. The fixed evaluator models use their original precision defaults: Whisper FP16 and the other evaluators FP32, identically for both synthesis variants. Both ASR metrics apply Whisper's `EnglishTextNormalizer` to the transcript and reference text, including its spelling and text normalization rules. The normalizer is more than punctuation removal.
152
+
153
+ | Metric | Fixed evaluator | Evaluator precision | Comparison |
154
+ |---|---|---|---|
155
+ | WER Whisper | `openai/whisper-large-v3` | FP16 | Target transcript, `EnglishTextNormalizer` |
156
+ | WER wav2vec2 | `facebook/wav2vec2-large-960h-lv60-self` | FP32 | Target transcript, the same `EnglishTextNormalizer` |
157
+ | SECS Resemblyzer | `resemble-ai/Resemblyzer` pretrained encoder at commit `15d828edebe06bc72b9cabc8ef8ca5ab2cb457ce` | FP32 | Prompt reference audio |
158
+ | SECS WavLM | `microsoft/wavlm-base-sv` at revision `0a23162ffc49adcf42bdf836a00cb2eb45af3601` | FP32 | Prompt reference audio |
159
+ | ECA | `emotion2vec/emotion2vec_plus_base` at revision `b318240bfe67db81a8c572ecb37ce9c3759b81c9`, restricted to the five ESD emotion logits | FP32 | Specified target emotion |
160
+ | EECS | The same emotion2vec+ base, mean frame embedding | FP32 | Prompt reference audio |
161
+ | SVAS | `audeering/wav2vec2-large-robust-12-ft-emotion-msp-dim` | FP32 | Ground-truth target audio |
162
+
163
  #### Seen speakers, 1,200 pairs
164
 
165
  | Precision | WER Whisper (%) ↓ | WER wav2vec2 (%) ↓ | WER avg (%) ↓ | SECS Resemblyzer ↑ | SECS WavLM ↑ | SECS avg ↑ | ECA (%) ↑ | EECS ↑ | SVAS ↑ |
 
197
 
198
  The BF16 estimator timestep input is part of the matched runtime contract. The FP32 scalar integration schedule is retained as solver metadata, without an FP32 inference-weight copy. No controlled inference-speed comparison is reported.
199
 
200
+ #### Repeatability qualification
201
+
202
+ A bounded probe on real ESD clip `0017_001436`, seed 1100000, 100 Euler steps and core batch size one found that repeated inference with unchanged inputs, noise and weights did not always produce bit-identical mels in either BF16 or H64. The responsible kernel has not been identified. The benchmark retains the first canonical generated output for each row, without selecting a repeat for better scores. Its paired sentence-cluster intervals do not separately estimate numerical repeat variability. No deterministic repeatability guarantee is claimed.
203
+
204
  ### Export verification
205
 
206
  A fresh `python -I` process loaded only `AutoProcessor` and `AutoModel` from this repository at code/weight revision `39f692e1ca0fc42b0d29589c946aa3aaed265f39`, with no project-package import or generation hooks. For ESD seen clip `0014_001095` and unseen clip `0013_001433`, processor tensors, generated mels and postprocessed waveforms exactly matched the saved strict BF16/H64 reference pipeline, including the lazy internal BF16 BigVGAN. Verification seeds 777700 and 777701 used stride-preserving `randn_like` noise; this two-clip parity check uses different noise from the full benchmark and establishes export parity, not quality equivalence. Packed state matched the frozen benchmark before and after generation. No optimizer step was performed.
207
 
208
+ That exact two-clip match describes this verified execution. It is separate from the full benchmark and does not guarantee bit-identical output across repeated executions.
209
+
210
  ### Paper context
211
 
212
  The following published EmoSphere++ results are from Table II (seen speakers) and Table III (unseen speakers) of [Cho et al.](https://ieeexplore.ieee.org/document/10965917), using the paper's ESD protocol, vocoder and evaluation setup. They provide source context separately from this conversion's matched BF16/H64 results. Different vocoder weights and evaluation details prevent treating the paper rows as matched comparisons to this conversion.