Text-to-Speech
Transformers
Safetensors
English
emosphere_pp
feature-extraction
emotional-tts
zero-shot-tts
flow-matching
emosphere
custom_code
Instructions to use RetentionLabs/EmoSphere-PP-16kHz with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use RetentionLabs/EmoSphere-PP-16kHz with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-to-speech", model="RetentionLabs/EmoSphere-PP-16kHz", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("RetentionLabs/EmoSphere-PP-16kHz", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
docs(model): report matched full ESD benchmark and paired uncertainty
Browse files
README.md
CHANGED
|
@@ -21,9 +21,10 @@ datasets:
|
|
| 21 |
|
| 22 |
EmoSphere++ is an emotion controllable zero shot text to speech model by Cho et al. (IEEE Transactions on Affective Computing, 2025). It synthesizes a 16 kHz, 80 band log mel spectrogram from English text, a reference recording of the target speaker and an emotion given as a point in a sphere: the radius sets the intensity, the elevation and azimuth set the style.
|
| 23 |
|
| 24 |
-
This
|
|
|
|
|
|
|
| 25 |
|
| 26 |
-
**Release status: private preflight candidate, final quality evaluation pending.** The model file matches this 42-linear H64 configuration. The earlier 0.47% candidate is backed up outside the upload directory with its SHA-256 in the audit record. The H64 candidate has not been publicly released. The repository-ID example below requires authorized access to this private repository. Public release remains subject to review of the matched full 1,500-pair quality evaluation. Technical export parity does not establish quality equivalence to BF16.
|
| 27 |
The model returns a 16 kHz waveform by default. It internally renders the mel with the frozen vocoder [`RetentionLabs/BigVGAN-16kHz-80band`](https://huggingface.co/RetentionLabs/BigVGAN-16kHz-80band).
|
| 28 |
|
| 29 |
- Paper: [doi:10.1109/TAFFC.2025.3561267](https://ieeexplore.ieee.org/document/10965917)
|
|
@@ -37,9 +38,9 @@ The model returns a 16 kHz waveform by default. It internally renders the mel wi
|
|
| 37 |
| Input | Phonemes (CMUdict and g2p_en), WavLM x-vector, emotion2vec+ base embedding, emotion-adaptive spherical vector $(r, \vartheta, \phi)$ |
|
| 38 |
| Output | 16 kHz waveform; optional 80-band log mel with `output_audio=False` |
|
| 39 |
| Acoustic core parameters | 28,183,009, excluding reference encoders and vocoder |
|
| 40 |
-
|
|
| 41 |
| Reference encoders and vocoder | BF16 weight parameters and floating model inputs, downloaded separately |
|
| 42 |
-
|
|
| 43 |
| Training data | ESD English (10 speakers, 5 emotions), by the authors |
|
| 44 |
|
| 45 |
### Storage scope
|
|
@@ -47,7 +48,7 @@ The model returns a 16 kHz waveform by default. It internally renders the mel wi
|
|
| 47 |
| Acoustic core from the released 1.1M-step checkpoint | Bytes | Decimal MB | Accounting |
|
| 48 |
|---|---:|---:|---|
|
| 49 |
| Logical BF16 parameter baseline | 56,366,018 | 56.366 | 28,183,009 parameters times two bytes, excluding serialization and buffers |
|
| 50 |
-
|
|
| 51 |
|
| 52 |
The logical BF16 row is a parameter-size calculation, rather than a measured BF16 safetensors file. These rows use different metadata accounting. Both exclude the separately downloaded reference encoders, BigVGAN, source files and runtime activations. The 19.533% selection describes the acoustic core only; the complete synthesis pipeline also contains the BF16 reference encoders and vocoder. No inference speedup, total-pipeline compression ratio or VRAM reduction is claimed.
|
| 53 |
|
|
@@ -65,7 +66,7 @@ The processor builds every model input from a reference recording: phoneme ids f
|
|
| 65 |
|
| 66 |
## Requirements
|
| 67 |
|
| 68 |
-
The following versions were observed in the existing environment used for the
|
| 69 |
|
| 70 |
| Component | Observed version |
|
| 71 |
|---|---|
|
|
@@ -100,11 +101,12 @@ import torch
|
|
| 100 |
from transformers import AutoModel, AutoProcessor
|
| 101 |
|
| 102 |
repo_id = "RetentionLabs/EmoSphere-PP-16kHz-NVFP4"
|
|
|
|
| 103 |
processor = AutoProcessor.from_pretrained(
|
| 104 |
-
repo_id, trust_remote_code=True, device="cuda", dtype="bfloat16",
|
| 105 |
)
|
| 106 |
model = AutoModel.from_pretrained(
|
| 107 |
-
repo_id, trust_remote_code=True, dtype=torch.bfloat16,
|
| 108 |
).to("cuda", dtype=torch.bfloat16).eval()
|
| 109 |
|
| 110 |
inputs = processor(
|
|
@@ -121,7 +123,7 @@ Only the processor and acoustic model need explicit loading. Generation loads an
|
|
| 121 |
|
| 122 |
The internal vocoder loader passes `dtype=torch.bfloat16` explicitly and stays pinned to BigVGAN revision `d012acc64091410e7c56fd135d6ec04ea5c11ac7`, matching the frozen quality benchmark. The public BigVGAN main revision `c01b64cf5a9328df6a9cd86e0390437f5b7d94c7` sets its default configuration to BF16 without changing the weights. In the installed Transformers 5.19.0, an omitted `dtype` is treated as `"auto"` and follows the saved configuration. The explicit BF16 argument keeps this candidate's runtime precision independent of vocoder metadata changes.
|
| 123 |
|
| 124 |
-
## Evaluation
|
| 125 |
|
| 126 |
### Adaptive selection on 240 recordings
|
| 127 |
|
|
@@ -138,20 +140,56 @@ The following rows use the authors' 1.1M-step acoustic checkpoint, the same BF16
|
|
| 138 |
|
| 139 |
On this selection set, the H64 candidate has lower seen-speaker WER alongside lower seen-speaker speaker and emotion readings. Its unseen-speaker WER is higher. These mixed changes and the small, repeatedly used selection set do not establish equivalence or a general improvement.
|
| 140 |
|
| 141 |
-
### Full
|
| 142 |
|
| 143 |
-
|
| 144 |
|
| 145 |
-
|
| 146 |
-
|
| 147 |
-
|
| 148 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 149 |
|
| 150 |
-
|
| 151 |
|
| 152 |
### Paper context
|
| 153 |
|
| 154 |
-
The following published EmoSphere++ results are from Table II (seen speakers) and Table III (unseen speakers) of [Cho et al.](https://ieeexplore.ieee.org/document/10965917), using the paper's ESD protocol, vocoder and evaluation setup. They provide source context separately from
|
| 155 |
|
| 156 |
| Published model and speaker group | WER avg (%) β | SECS avg β | ECA (%) β | SVAS β | EECS β |
|
| 157 |
|---|---:|---:|---:|---:|---:|
|
|
@@ -173,7 +211,7 @@ The normalized Hadamard, rotation and activation quantization helper bodies are
|
|
| 173 |
## Limitations
|
| 174 |
|
| 175 |
- English only, trained on 10 ESD speakers. The selection rows cover ESD recording conditions; they do not establish quality on arbitrary voices or other recording conditions.
|
| 176 |
-
-
|
| 177 |
- The spherical vector is computed from the emotion label and the reference recording; an emotion other than the five ESD emotions is not supported.
|
| 178 |
- The text front end rejects numeric digits. Write numbers as English words before calling the processor.
|
| 179 |
- The vocoder is ours, not the authors'; results differ from the paper by that and by evaluation details.
|
|
|
|
| 21 |
|
| 22 |
EmoSphere++ is an emotion controllable zero shot text to speech model by Cho et al. (IEEE Transactions on Affective Computing, 2025). It synthesizes a 16 kHz, 80 band log mel spectrogram from English text, a reference recording of the target speaker and an emotion given as a point in a sphere: the radius sets the intensity, the elevation and azimuth set the style.
|
| 23 |
|
| 24 |
+
This repository provides an optional CFM-only NVFP4 conversion of the authors' released 1.1M-step checkpoint (`model_ckpt_steps_1100000.ckpt`). The candidate uses a normalized 64 x 64 Hadamard rotation and static round-to-nearest packing for 42 decoder linears. Their 5,505,024 weights account for 19.533% of the 28,183,009 acoustic-core parameters. This percentage excludes all reference encoders and the vocoder. Text, style, the timestep MLP and every nonquantized parameter use BF16. No QAT or other weight-update training is applied.
|
| 25 |
+
|
| 26 |
+
The matched 1,500-pair ESD comparison against BF16 is complete and reported below. This H64 conversion provides an optional mixed-precision variant. BF16 remains the default reference precision. The measurements show small tradeoffs on this fixed benchmark and do not establish population-level quality equivalence.
|
| 27 |
|
|
|
|
| 28 |
The model returns a 16 kHz waveform by default. It internally renders the mel with the frozen vocoder [`RetentionLabs/BigVGAN-16kHz-80band`](https://huggingface.co/RetentionLabs/BigVGAN-16kHz-80band).
|
| 29 |
|
| 30 |
- Paper: [doi:10.1109/TAFFC.2025.3561267](https://ieeexplore.ieee.org/document/10965917)
|
|
|
|
| 38 |
| Input | Phonemes (CMUdict and g2p_en), WavLM x-vector, emotion2vec+ base embedding, emotion-adaptive spherical vector $(r, \vartheta, \phi)$ |
|
| 39 |
| Output | 16 kHz waveform; optional 80-band log mel with `output_audio=False` |
|
| 40 |
| Acoustic core parameters | 28,183,009, excluding reference encoders and vocoder |
|
| 41 |
+
| Weight precision | Mixed NVFP4/BF16: 5,505,024 weights in 42 CFM linears stored in NVFP4, 19.533% of the acoustic-core parameter count; all other weight parameters BF16 |
|
| 42 |
| Reference encoders and vocoder | BF16 weight parameters and floating model inputs, downloaded separately |
|
| 43 |
+
| Acoustic-core weight file | `model.safetensors`, 48,502,914 bytes (48.503 MB, decimal) |
|
| 44 |
| Training data | ESD English (10 speakers, 5 emotions), by the authors |
|
| 45 |
|
| 46 |
### Storage scope
|
|
|
|
| 48 |
| Acoustic core from the released 1.1M-step checkpoint | Bytes | Decimal MB | Accounting |
|
| 49 |
|---|---:|---:|---|
|
| 50 |
| Logical BF16 parameter baseline | 56,366,018 | 56.366 | 28,183,009 parameters times two bytes, excluding serialization and buffers |
|
| 51 |
+
| H64 mixed NVFP4/BF16 `model.safetensors` | 48,502,914 | 48.503 | Actual file, including packed codes, scales, serialized buffers, recipe digest and safetensors header |
|
| 52 |
|
| 53 |
The logical BF16 row is a parameter-size calculation, rather than a measured BF16 safetensors file. These rows use different metadata accounting. Both exclude the separately downloaded reference encoders, BigVGAN, source files and runtime activations. The 19.533% selection describes the acoustic core only; the complete synthesis pipeline also contains the BF16 reference encoders and vocoder. No inference speedup, total-pipeline compression ratio or VRAM reduction is claimed.
|
| 54 |
|
|
|
|
| 66 |
|
| 67 |
## Requirements
|
| 68 |
|
| 69 |
+
The following versions were observed in the existing environment used for the matched benchmark and the two-clip Hub-revision GPU parity check. No fresh-install reproduction or compatibility across other version combinations has been verified.
|
| 70 |
|
| 71 |
| Component | Observed version |
|
| 72 |
|---|---|
|
|
|
|
| 101 |
from transformers import AutoModel, AutoProcessor
|
| 102 |
|
| 103 |
repo_id = "RetentionLabs/EmoSphere-PP-16kHz-NVFP4"
|
| 104 |
+
revision = "39f692e1ca0fc42b0d29589c946aa3aaed265f39" # verified code and weights
|
| 105 |
processor = AutoProcessor.from_pretrained(
|
| 106 |
+
repo_id, revision=revision, trust_remote_code=True, device="cuda", dtype="bfloat16",
|
| 107 |
)
|
| 108 |
model = AutoModel.from_pretrained(
|
| 109 |
+
repo_id, revision=revision, trust_remote_code=True, dtype=torch.bfloat16,
|
| 110 |
).to("cuda", dtype=torch.bfloat16).eval()
|
| 111 |
|
| 112 |
inputs = processor(
|
|
|
|
| 123 |
|
| 124 |
The internal vocoder loader passes `dtype=torch.bfloat16` explicitly and stays pinned to BigVGAN revision `d012acc64091410e7c56fd135d6ec04ea5c11ac7`, matching the frozen quality benchmark. The public BigVGAN main revision `c01b64cf5a9328df6a9cd86e0390437f5b7d94c7` sets its default configuration to BF16 without changing the weights. In the installed Transformers 5.19.0, an omitted `dtype` is treated as `"auto"` and follows the saved configuration. The explicit BF16 argument keeps this candidate's runtime precision independent of vocoder metadata changes.
|
| 125 |
|
| 126 |
+
## Evaluation
|
| 127 |
|
| 128 |
### Adaptive selection on 240 recordings
|
| 129 |
|
|
|
|
| 140 |
|
| 141 |
On this selection set, the H64 candidate has lower seen-speaker WER alongside lower seen-speaker speaker and emotion readings. Its unseen-speaker WER is higher. These mixed changes and the small, repeatedly used selection set do not establish equivalence or a general improvement.
|
| 142 |
|
| 143 |
+
### Full matched ESD benchmark
|
| 144 |
|
| 145 |
+
The authors' full ESD protocol comprises 1,200 seen-speaker pairs and 300 unseen-speaker pairs. Both rows use the released 1.1M-step acoustic checkpoint, the same BF16 conditioning features and per-pair common noise, 100 Euler steps, core batch size one, and the same 40k-step BigVGAN in BF16 with vocoder batch size 32. The H64 weights are packed once, remain unchanged during generation, and receive no training. Metric implementations are shared across both rows; their precision is separate from synthesis precision. WER is corpus-level, and each avg column is the arithmetic mean of its two named evaluators.
|
| 146 |
|
| 147 |
+
Results and setup: [matched full ESD benchmark on W&B](https://wandb.ai/haan_modeling/vadweaver/runs/6958b357).
|
| 148 |
+
|
| 149 |
+
#### Seen speakers, 1,200 pairs
|
| 150 |
+
|
| 151 |
+
| Precision | WER Whisper (%) β | WER wav2vec2 (%) β | WER avg (%) β | SECS Resemblyzer β | SECS WavLM β | SECS avg β | ECA (%) β | EECS β | SVAS β |
|
| 152 |
+
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|
| 153 |
+
| BF16 reference | 9.172 | 11.428 | 10.300 | 0.74907 | 0.92472 | 0.83689 | 92.917 | 0.94469 | 0.90656 |
|
| 154 |
+
| H64 mixed NVFP4/BF16 | 9.115 | 11.336 | 10.226 | 0.74539 | 0.92369 | 0.83454 | 93.083 | 0.94821 | 0.91031 |
|
| 155 |
+
|
| 156 |
+
#### Unseen speakers, 300 pairs
|
| 157 |
+
|
| 158 |
+
| Precision | WER Whisper (%) β | WER wav2vec2 (%) β | WER avg (%) β | SECS Resemblyzer β | SECS WavLM β | SECS avg β | ECA (%) β | EECS β | SVAS β |
|
| 159 |
+
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|
| 160 |
+
| BF16 reference | 7.755 | 9.580 | 8.668 | 0.64703 | 0.89293 | 0.76998 | 91.000 | 0.90314 | 0.85792 |
|
| 161 |
+
| H64 mixed NVFP4/BF16 | 7.938 | 9.672 | 8.805 | 0.64536 | 0.89038 | 0.76787 | 91.000 | 0.91393 | 0.88300 |
|
| 162 |
+
|
| 163 |
+
#### Paired differences and confidence intervals
|
| 164 |
+
|
| 165 |
+
Differences are H64 minus BF16. WER and ECA differences use percentage points; similarity metrics are dimensionless. The 95% intervals use 10,000 paired bootstrap resamples by the 30 parallel ESD sentence positions within each speaker group, conditional on the fixed speakers. They cover the five aggregate metrics below; no intervals for the individual ASR or speaker evaluators are inferred.
|
| 166 |
+
|
| 167 |
+
| Speaker group | Metric | Difference | Paired 95% CI |
|
| 168 |
+
|---|---|---:|---|
|
| 169 |
+
| Seen | WER avg (pp) | -0.074 | [-0.288, 0.162] |
|
| 170 |
+
| Seen | SECS avg | -0.002357 | [-0.002870, -0.001841] |
|
| 171 |
+
| Seen | ECA (pp) | +0.167 | [-0.583, 0.917] |
|
| 172 |
+
| Seen | EECS | +0.003514 | [0.000334, 0.006857] |
|
| 173 |
+
| Seen | SVAS | +0.003741 | [-0.004099, 0.012922] |
|
| 174 |
+
| Unseen | WER avg (pp) | +0.137 | [-0.330, 0.567] |
|
| 175 |
+
| Unseen | SECS avg | -0.002109 | [-0.003227, -0.001027] |
|
| 176 |
+
| Unseen | ECA (pp) | +0.000 | [-1.333, 1.333] |
|
| 177 |
+
| Unseen | EECS | +0.010793 | [0.000101, 0.020127] |
|
| 178 |
+
| Unseen | SVAS | +0.025080 | [0.006190, 0.049711] |
|
| 179 |
+
|
| 180 |
+
H64 changes WER avg by -0.074 percentage points on seen speakers and +0.137 on unseen speakers. ECA changes by +0.167 points on seen speakers and zero on unseen speakers. SECS avg decreases by 0.002357 and 0.002109 respectively, with negative paired intervals in both groups. EECS and SVAS point estimates increase. These are measured tradeoffs of the optional H64 variant on this benchmark; they do not establish a population-level equivalence margin or a uniform quality improvement.
|
| 181 |
+
|
| 182 |
+
Exact utterance overlap between the 240-recording selection and the full benchmark is zero for selection-target/benchmark-target, selection-target/benchmark-reference, selection-reference/benchmark-target and selection-reference/benchmark-reference comparisons. However, all 1,500 benchmark target recordings were evaluated with earlier FP32/NVFP4 configurations during the historical search. The full benchmark is therefore not an untouched holdout. Its paired intervals are conditional on these fixed speakers and sentence positions; they do not account for adaptive selection or generalize uncertainty to arbitrary speakers.
|
| 183 |
+
|
| 184 |
+
The BF16 estimator timestep input is part of the matched runtime contract. The FP32 scalar integration schedule is retained as solver metadata, without an FP32 inference-weight copy. No controlled inference-speed comparison is reported.
|
| 185 |
+
|
| 186 |
+
### Export verification
|
| 187 |
|
| 188 |
+
A fresh `python -I` process loaded only `AutoProcessor` and `AutoModel` from this repository at code/weight revision `39f692e1ca0fc42b0d29589c946aa3aaed265f39`, with no project-package import or generation hooks. For ESD seen clip `0014_001095` and unseen clip `0013_001433`, processor tensors, generated mels and postprocessed waveforms exactly matched the saved strict BF16/H64 reference pipeline, including the lazy internal BF16 BigVGAN. Verification seeds 777700 and 777701 used stride-preserving `randn_like` noise; this two-clip parity check uses different noise from the full benchmark and establishes export parity, not quality equivalence. Packed state matched the frozen benchmark before and after generation. No optimizer step was performed.
|
| 189 |
|
| 190 |
### Paper context
|
| 191 |
|
| 192 |
+
The following published EmoSphere++ results are from Table II (seen speakers) and Table III (unseen speakers) of [Cho et al.](https://ieeexplore.ieee.org/document/10965917), using the paper's ESD protocol, vocoder and evaluation setup. They provide source context separately from this conversion's matched BF16/H64 results. Different vocoder weights and evaluation details prevent treating the paper rows as matched comparisons to this conversion.
|
| 193 |
|
| 194 |
| Published model and speaker group | WER avg (%) β | SECS avg β | ECA (%) β | SVAS β | EECS β |
|
| 195 |
|---|---:|---:|---:|---:|---:|
|
|
|
|
| 211 |
## Limitations
|
| 212 |
|
| 213 |
- English only, trained on 10 ESD speakers. The selection rows cover ESD recording conditions; they do not establish quality on arbitrary voices or other recording conditions.
|
| 214 |
+
- The full ESD comparison is conditional on a historically inspected benchmark and fixed speakers. No formal population-level quality equivalence or inference speedup is claimed.
|
| 215 |
- The spherical vector is computed from the emotion label and the reference recording; an emotion other than the five ESD emotions is not supported.
|
| 216 |
- The text front end rejects numeric digits. Write numbers as English words before calling the processor.
|
| 217 |
- The vocoder is ours, not the authors'; results differ from the paper by that and by evaluation details.
|