Text-to-Speech
Transformers
Safetensors
English
emosphere_pp
feature-extraction
emotional-tts
zero-shot-tts
flow-matching
emosphere
custom_code
Instructions to use RetentionLabs/EmoSphere-PP-16kHz with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use RetentionLabs/EmoSphere-PP-16kHz with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-to-speech", model="RetentionLabs/EmoSphere-PP-16kHz", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("RetentionLabs/EmoSphere-PP-16kHz", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
docs(model): specify evaluator precision and bounded repeatability
Browse files
README.md
CHANGED
|
@@ -146,6 +146,20 @@ The authors' full ESD protocol comprises 1,200 seen-speaker pairs and 300 unseen
|
|
| 146 |
|
| 147 |
Results and setup: [matched full ESD benchmark on W&B](https://wandb.ai/haan_modeling/vadweaver/runs/6958b357).
|
| 148 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 149 |
#### Seen speakers, 1,200 pairs
|
| 150 |
|
| 151 |
| Precision | WER Whisper (%) β | WER wav2vec2 (%) β | WER avg (%) β | SECS Resemblyzer β | SECS WavLM β | SECS avg β | ECA (%) β | EECS β | SVAS β |
|
|
@@ -183,10 +197,16 @@ Exact utterance overlap between the 240-recording selection and the full benchma
|
|
| 183 |
|
| 184 |
The BF16 estimator timestep input is part of the matched runtime contract. The FP32 scalar integration schedule is retained as solver metadata, without an FP32 inference-weight copy. No controlled inference-speed comparison is reported.
|
| 185 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 186 |
### Export verification
|
| 187 |
|
| 188 |
A fresh `python -I` process loaded only `AutoProcessor` and `AutoModel` from this repository at code/weight revision `39f692e1ca0fc42b0d29589c946aa3aaed265f39`, with no project-package import or generation hooks. For ESD seen clip `0014_001095` and unseen clip `0013_001433`, processor tensors, generated mels and postprocessed waveforms exactly matched the saved strict BF16/H64 reference pipeline, including the lazy internal BF16 BigVGAN. Verification seeds 777700 and 777701 used stride-preserving `randn_like` noise; this two-clip parity check uses different noise from the full benchmark and establishes export parity, not quality equivalence. Packed state matched the frozen benchmark before and after generation. No optimizer step was performed.
|
| 189 |
|
|
|
|
|
|
|
| 190 |
### Paper context
|
| 191 |
|
| 192 |
The following published EmoSphere++ results are from Table II (seen speakers) and Table III (unseen speakers) of [Cho et al.](https://ieeexplore.ieee.org/document/10965917), using the paper's ESD protocol, vocoder and evaluation setup. They provide source context separately from this conversion's matched BF16/H64 results. Different vocoder weights and evaluation details prevent treating the paper rows as matched comparisons to this conversion.
|
|
|
|
| 146 |
|
| 147 |
Results and setup: [matched full ESD benchmark on W&B](https://wandb.ai/haan_modeling/vadweaver/runs/6958b357).
|
| 148 |
|
| 149 |
+
#### Metric protocol
|
| 150 |
+
|
| 151 |
+
Synthesis uses BF16 for unquantized parameters, reference encoders, floating model inputs and BigVGAN. The fixed evaluator models use their original precision defaults: Whisper FP16 and the other evaluators FP32, identically for both synthesis variants. Both ASR metrics apply Whisper's `EnglishTextNormalizer` to the transcript and reference text, including its spelling and text normalization rules. The normalizer is more than punctuation removal.
|
| 152 |
+
|
| 153 |
+
| Metric | Fixed evaluator | Evaluator precision | Comparison |
|
| 154 |
+
|---|---|---|---|
|
| 155 |
+
| WER Whisper | `openai/whisper-large-v3` | FP16 | Target transcript, `EnglishTextNormalizer` |
|
| 156 |
+
| WER wav2vec2 | `facebook/wav2vec2-large-960h-lv60-self` | FP32 | Target transcript, the same `EnglishTextNormalizer` |
|
| 157 |
+
| SECS Resemblyzer | `resemble-ai/Resemblyzer` pretrained encoder at commit `15d828edebe06bc72b9cabc8ef8ca5ab2cb457ce` | FP32 | Prompt reference audio |
|
| 158 |
+
| SECS WavLM | `microsoft/wavlm-base-sv` at revision `0a23162ffc49adcf42bdf836a00cb2eb45af3601` | FP32 | Prompt reference audio |
|
| 159 |
+
| ECA | `emotion2vec/emotion2vec_plus_base` at revision `b318240bfe67db81a8c572ecb37ce9c3759b81c9`, restricted to the five ESD emotion logits | FP32 | Specified target emotion |
|
| 160 |
+
| EECS | The same emotion2vec+ base, mean frame embedding | FP32 | Prompt reference audio |
|
| 161 |
+
| SVAS | `audeering/wav2vec2-large-robust-12-ft-emotion-msp-dim` | FP32 | Ground-truth target audio |
|
| 162 |
+
|
| 163 |
#### Seen speakers, 1,200 pairs
|
| 164 |
|
| 165 |
| Precision | WER Whisper (%) β | WER wav2vec2 (%) β | WER avg (%) β | SECS Resemblyzer β | SECS WavLM β | SECS avg β | ECA (%) β | EECS β | SVAS β |
|
|
|
|
| 197 |
|
| 198 |
The BF16 estimator timestep input is part of the matched runtime contract. The FP32 scalar integration schedule is retained as solver metadata, without an FP32 inference-weight copy. No controlled inference-speed comparison is reported.
|
| 199 |
|
| 200 |
+
#### Repeatability qualification
|
| 201 |
+
|
| 202 |
+
A bounded probe on real ESD clip `0017_001436`, seed 1100000, 100 Euler steps and core batch size one found that repeated inference with unchanged inputs, noise and weights did not always produce bit-identical mels in either BF16 or H64. The responsible kernel has not been identified. The benchmark retains the first canonical generated output for each row, without selecting a repeat for better scores. Its paired sentence-cluster intervals do not separately estimate numerical repeat variability. No deterministic repeatability guarantee is claimed.
|
| 203 |
+
|
| 204 |
### Export verification
|
| 205 |
|
| 206 |
A fresh `python -I` process loaded only `AutoProcessor` and `AutoModel` from this repository at code/weight revision `39f692e1ca0fc42b0d29589c946aa3aaed265f39`, with no project-package import or generation hooks. For ESD seen clip `0014_001095` and unseen clip `0013_001433`, processor tensors, generated mels and postprocessed waveforms exactly matched the saved strict BF16/H64 reference pipeline, including the lazy internal BF16 BigVGAN. Verification seeds 777700 and 777701 used stride-preserving `randn_like` noise; this two-clip parity check uses different noise from the full benchmark and establishes export parity, not quality equivalence. Packed state matched the frozen benchmark before and after generation. No optimizer step was performed.
|
| 207 |
|
| 208 |
+
That exact two-clip match describes this verified execution. It is separate from the full benchmark and does not guarantee bit-identical output across repeated executions.
|
| 209 |
+
|
| 210 |
### Paper context
|
| 211 |
|
| 212 |
The following published EmoSphere++ results are from Table II (seen speakers) and Table III (unseen speakers) of [Cho et al.](https://ieeexplore.ieee.org/document/10965917), using the paper's ESD protocol, vocoder and evaluation setup. They provide source context separately from this conversion's matched BF16/H64 results. Different vocoder weights and evaluation details prevent treating the paper rows as matched comparisons to this conversion.
|