RetentionLabs commited on
Commit
40d6c52
Β·
verified Β·
1 Parent(s): 39f692e

docs(model): report matched full ESD benchmark and paired uncertainty

Browse files
Files changed (1) hide show
  1. README.md +56 -18
README.md CHANGED
@@ -21,9 +21,10 @@ datasets:
21
 
22
  EmoSphere++ is an emotion controllable zero shot text to speech model by Cho et al. (IEEE Transactions on Affective Computing, 2025). It synthesizes a 16 kHz, 80 band log mel spectrogram from English text, a reference recording of the target speaker and an emotion given as a point in a sphere: the radius sets the intensity, the elevation and azimuth set the style.
23
 
24
- This private preflight repository contains a CFM-only NVFP4 conversion of the authors' released 1.1M-step checkpoint (`model_ckpt_steps_1100000.ckpt`). The candidate uses a normalized 64 x 64 Hadamard rotation and static round-to-nearest packing for 42 decoder linears. Their 5,505,024 weights account for 19.533% of the 28,183,009 acoustic-core parameters. This percentage excludes all reference encoders and the vocoder. Text, style, the timestep MLP and every nonquantized parameter use BF16. No QAT or other weight-update training is applied.
 
 
25
 
26
- **Release status: private preflight candidate, final quality evaluation pending.** The model file matches this 42-linear H64 configuration. The earlier 0.47% candidate is backed up outside the upload directory with its SHA-256 in the audit record. The H64 candidate has not been publicly released. The repository-ID example below requires authorized access to this private repository. Public release remains subject to review of the matched full 1,500-pair quality evaluation. Technical export parity does not establish quality equivalence to BF16.
27
  The model returns a 16 kHz waveform by default. It internally renders the mel with the frozen vocoder [`RetentionLabs/BigVGAN-16kHz-80band`](https://huggingface.co/RetentionLabs/BigVGAN-16kHz-80band).
28
 
29
  - Paper: [doi:10.1109/TAFFC.2025.3561267](https://ieeexplore.ieee.org/document/10965917)
@@ -37,9 +38,9 @@ The model returns a 16 kHz waveform by default. It internally renders the mel wi
37
  | Input | Phonemes (CMUdict and g2p_en), WavLM x-vector, emotion2vec+ base embedding, emotion-adaptive spherical vector $(r, \vartheta, \phi)$ |
38
  | Output | 16 kHz waveform; optional 80-band log mel with `output_audio=False` |
39
  | Acoustic core parameters | 28,183,009, excluding reference encoders and vocoder |
40
- | Candidate precision | Mixed NVFP4/BF16: 5,505,024 weights in 42 CFM linears stored in NVFP4, 19.533% of the acoustic-core parameter count; all other weight parameters BF16 |
41
  | Reference encoders and vocoder | BF16 weight parameters and floating model inputs, downloaded separately |
42
- | Local acoustic-core weight file | `model.safetensors`, 48,502,914 bytes (48.503 MB, decimal) |
43
  | Training data | ESD English (10 speakers, 5 emotions), by the authors |
44
 
45
  ### Storage scope
@@ -47,7 +48,7 @@ The model returns a 16 kHz waveform by default. It internally renders the mel wi
47
  | Acoustic core from the released 1.1M-step checkpoint | Bytes | Decimal MB | Accounting |
48
  |---|---:|---:|---|
49
  | Logical BF16 parameter baseline | 56,366,018 | 56.366 | 28,183,009 parameters times two bytes, excluding serialization and buffers |
50
- | Local H64 mixed NVFP4/BF16 `model.safetensors` | 48,502,914 | 48.503 | Actual file, including packed codes, scales, serialized buffers, recipe digest and safetensors header |
51
 
52
  The logical BF16 row is a parameter-size calculation, rather than a measured BF16 safetensors file. These rows use different metadata accounting. Both exclude the separately downloaded reference encoders, BigVGAN, source files and runtime activations. The 19.533% selection describes the acoustic core only; the complete synthesis pipeline also contains the BF16 reference encoders and vocoder. No inference speedup, total-pipeline compression ratio or VRAM reduction is claimed.
53
 
@@ -65,7 +66,7 @@ The processor builds every model input from a reference recording: phoneme ids f
65
 
66
  ## Requirements
67
 
68
- The following versions were observed in the existing environment used for the local two-clip GPU parity check. No fresh-install reproduction or compatibility across other version combinations has been verified.
69
 
70
  | Component | Observed version |
71
  |---|---|
@@ -100,11 +101,12 @@ import torch
100
  from transformers import AutoModel, AutoProcessor
101
 
102
  repo_id = "RetentionLabs/EmoSphere-PP-16kHz-NVFP4"
 
103
  processor = AutoProcessor.from_pretrained(
104
- repo_id, trust_remote_code=True, device="cuda", dtype="bfloat16",
105
  )
106
  model = AutoModel.from_pretrained(
107
- repo_id, trust_remote_code=True, dtype=torch.bfloat16,
108
  ).to("cuda", dtype=torch.bfloat16).eval()
109
 
110
  inputs = processor(
@@ -121,7 +123,7 @@ Only the processor and acoustic model need explicit loading. Generation loads an
121
 
122
  The internal vocoder loader passes `dtype=torch.bfloat16` explicitly and stays pinned to BigVGAN revision `d012acc64091410e7c56fd135d6ec04ea5c11ac7`, matching the frozen quality benchmark. The public BigVGAN main revision `c01b64cf5a9328df6a9cd86e0390437f5b7d94c7` sets its default configuration to BF16 without changing the weights. In the installed Transformers 5.19.0, an omitted `dtype` is treated as `"auto"` and follows the saved configuration. The explicit BF16 argument keeps this candidate's runtime precision independent of vocoder metadata changes.
123
 
124
- ## Evaluation status
125
 
126
  ### Adaptive selection on 240 recordings
127
 
@@ -138,20 +140,56 @@ The following rows use the authors' 1.1M-step acoustic checkpoint, the same BF16
138
 
139
  On this selection set, the H64 candidate has lower seen-speaker WER alongside lower seen-speaker speaker and emotion readings. Its unseen-speaker WER is higher. These mixed changes and the small, repeatedly used selection set do not establish equivalence or a general improvement.
140
 
141
- ### Full benchmark, pending
142
 
143
- A matched full 1,500-pair comparison against BF16 is pending, covering 1,200 seen-speaker and 300 unseen-speaker pairs. No final H64 WER, speaker similarity or emotion accuracy is claimed here. Earlier FP32 and 0.47% candidate tables are preserved in the local audit record and do not describe this candidate. The ESD test protocol has been inspected during earlier searches, so the full benchmark must also disclose that evaluation history.
144
 
145
- | Full ESD benchmark | BF16 reference | H64 mixed NVFP4/BF16 |
146
- |---|---|---|
147
- | Seen speakers, 1,200 pairs | Pending final integration | Pending final integration |
148
- | Unseen speakers, 300 pairs | Pending final integration | Pending final integration |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
149
 
150
- The final comparison must use the same BF16 reference features, per-pair noise, Euler schedule, inference batch grouping, 40k-step BigVGAN and metric setup for both candidates. The BF16 estimator timestep input is part of that contract. The FP32 scalar integration schedule is retained; it is not a model parameter or an FP32 weight copy.
151
 
152
  ### Paper context
153
 
154
- The following published EmoSphere++ results are from Table II (seen speakers) and Table III (unseen speakers) of [Cho et al.](https://ieeexplore.ieee.org/document/10965917), using the paper's ESD protocol, vocoder and evaluation setup. They provide source context separately from the adaptive 240-recording selection above. Different dataset sizes, vocoder weights and evaluation details prevent treating those rows as matched comparisons to the staged candidate.
155
 
156
  | Published model and speaker group | WER avg (%) ↓ | SECS avg ↑ | ECA (%) ↑ | SVAS ↑ | EECS ↑ |
157
  |---|---:|---:|---:|---:|---:|
@@ -173,7 +211,7 @@ The normalized Hadamard, rotation and activation quantization helper bodies are
173
  ## Limitations
174
 
175
  - English only, trained on 10 ESD speakers. The selection rows cover ESD recording conditions; they do not establish quality on arbitrary voices or other recording conditions.
176
- - Final quality equivalence to BF16 is pending. No controlled inference speed benchmark is reported.
177
  - The spherical vector is computed from the emotion label and the reference recording; an emotion other than the five ESD emotions is not supported.
178
  - The text front end rejects numeric digits. Write numbers as English words before calling the processor.
179
  - The vocoder is ours, not the authors'; results differ from the paper by that and by evaluation details.
 
21
 
22
  EmoSphere++ is an emotion controllable zero shot text to speech model by Cho et al. (IEEE Transactions on Affective Computing, 2025). It synthesizes a 16 kHz, 80 band log mel spectrogram from English text, a reference recording of the target speaker and an emotion given as a point in a sphere: the radius sets the intensity, the elevation and azimuth set the style.
23
 
24
+ This repository provides an optional CFM-only NVFP4 conversion of the authors' released 1.1M-step checkpoint (`model_ckpt_steps_1100000.ckpt`). The candidate uses a normalized 64 x 64 Hadamard rotation and static round-to-nearest packing for 42 decoder linears. Their 5,505,024 weights account for 19.533% of the 28,183,009 acoustic-core parameters. This percentage excludes all reference encoders and the vocoder. Text, style, the timestep MLP and every nonquantized parameter use BF16. No QAT or other weight-update training is applied.
25
+
26
+ The matched 1,500-pair ESD comparison against BF16 is complete and reported below. This H64 conversion provides an optional mixed-precision variant. BF16 remains the default reference precision. The measurements show small tradeoffs on this fixed benchmark and do not establish population-level quality equivalence.
27
 
 
28
  The model returns a 16 kHz waveform by default. It internally renders the mel with the frozen vocoder [`RetentionLabs/BigVGAN-16kHz-80band`](https://huggingface.co/RetentionLabs/BigVGAN-16kHz-80band).
29
 
30
  - Paper: [doi:10.1109/TAFFC.2025.3561267](https://ieeexplore.ieee.org/document/10965917)
 
38
  | Input | Phonemes (CMUdict and g2p_en), WavLM x-vector, emotion2vec+ base embedding, emotion-adaptive spherical vector $(r, \vartheta, \phi)$ |
39
  | Output | 16 kHz waveform; optional 80-band log mel with `output_audio=False` |
40
  | Acoustic core parameters | 28,183,009, excluding reference encoders and vocoder |
41
+ | Weight precision | Mixed NVFP4/BF16: 5,505,024 weights in 42 CFM linears stored in NVFP4, 19.533% of the acoustic-core parameter count; all other weight parameters BF16 |
42
  | Reference encoders and vocoder | BF16 weight parameters and floating model inputs, downloaded separately |
43
+ | Acoustic-core weight file | `model.safetensors`, 48,502,914 bytes (48.503 MB, decimal) |
44
  | Training data | ESD English (10 speakers, 5 emotions), by the authors |
45
 
46
  ### Storage scope
 
48
  | Acoustic core from the released 1.1M-step checkpoint | Bytes | Decimal MB | Accounting |
49
  |---|---:|---:|---|
50
  | Logical BF16 parameter baseline | 56,366,018 | 56.366 | 28,183,009 parameters times two bytes, excluding serialization and buffers |
51
+ | H64 mixed NVFP4/BF16 `model.safetensors` | 48,502,914 | 48.503 | Actual file, including packed codes, scales, serialized buffers, recipe digest and safetensors header |
52
 
53
  The logical BF16 row is a parameter-size calculation, rather than a measured BF16 safetensors file. These rows use different metadata accounting. Both exclude the separately downloaded reference encoders, BigVGAN, source files and runtime activations. The 19.533% selection describes the acoustic core only; the complete synthesis pipeline also contains the BF16 reference encoders and vocoder. No inference speedup, total-pipeline compression ratio or VRAM reduction is claimed.
54
 
 
66
 
67
  ## Requirements
68
 
69
+ The following versions were observed in the existing environment used for the matched benchmark and the two-clip Hub-revision GPU parity check. No fresh-install reproduction or compatibility across other version combinations has been verified.
70
 
71
  | Component | Observed version |
72
  |---|---|
 
101
  from transformers import AutoModel, AutoProcessor
102
 
103
  repo_id = "RetentionLabs/EmoSphere-PP-16kHz-NVFP4"
104
+ revision = "39f692e1ca0fc42b0d29589c946aa3aaed265f39" # verified code and weights
105
  processor = AutoProcessor.from_pretrained(
106
+ repo_id, revision=revision, trust_remote_code=True, device="cuda", dtype="bfloat16",
107
  )
108
  model = AutoModel.from_pretrained(
109
+ repo_id, revision=revision, trust_remote_code=True, dtype=torch.bfloat16,
110
  ).to("cuda", dtype=torch.bfloat16).eval()
111
 
112
  inputs = processor(
 
123
 
124
  The internal vocoder loader passes `dtype=torch.bfloat16` explicitly and stays pinned to BigVGAN revision `d012acc64091410e7c56fd135d6ec04ea5c11ac7`, matching the frozen quality benchmark. The public BigVGAN main revision `c01b64cf5a9328df6a9cd86e0390437f5b7d94c7` sets its default configuration to BF16 without changing the weights. In the installed Transformers 5.19.0, an omitted `dtype` is treated as `"auto"` and follows the saved configuration. The explicit BF16 argument keeps this candidate's runtime precision independent of vocoder metadata changes.
125
 
126
+ ## Evaluation
127
 
128
  ### Adaptive selection on 240 recordings
129
 
 
140
 
141
  On this selection set, the H64 candidate has lower seen-speaker WER alongside lower seen-speaker speaker and emotion readings. Its unseen-speaker WER is higher. These mixed changes and the small, repeatedly used selection set do not establish equivalence or a general improvement.
142
 
143
+ ### Full matched ESD benchmark
144
 
145
+ The authors' full ESD protocol comprises 1,200 seen-speaker pairs and 300 unseen-speaker pairs. Both rows use the released 1.1M-step acoustic checkpoint, the same BF16 conditioning features and per-pair common noise, 100 Euler steps, core batch size one, and the same 40k-step BigVGAN in BF16 with vocoder batch size 32. The H64 weights are packed once, remain unchanged during generation, and receive no training. Metric implementations are shared across both rows; their precision is separate from synthesis precision. WER is corpus-level, and each avg column is the arithmetic mean of its two named evaluators.
146
 
147
+ Results and setup: [matched full ESD benchmark on W&B](https://wandb.ai/haan_modeling/vadweaver/runs/6958b357).
148
+
149
+ #### Seen speakers, 1,200 pairs
150
+
151
+ | Precision | WER Whisper (%) ↓ | WER wav2vec2 (%) ↓ | WER avg (%) ↓ | SECS Resemblyzer ↑ | SECS WavLM ↑ | SECS avg ↑ | ECA (%) ↑ | EECS ↑ | SVAS ↑ |
152
+ |---|---:|---:|---:|---:|---:|---:|---:|---:|---:|
153
+ | BF16 reference | 9.172 | 11.428 | 10.300 | 0.74907 | 0.92472 | 0.83689 | 92.917 | 0.94469 | 0.90656 |
154
+ | H64 mixed NVFP4/BF16 | 9.115 | 11.336 | 10.226 | 0.74539 | 0.92369 | 0.83454 | 93.083 | 0.94821 | 0.91031 |
155
+
156
+ #### Unseen speakers, 300 pairs
157
+
158
+ | Precision | WER Whisper (%) ↓ | WER wav2vec2 (%) ↓ | WER avg (%) ↓ | SECS Resemblyzer ↑ | SECS WavLM ↑ | SECS avg ↑ | ECA (%) ↑ | EECS ↑ | SVAS ↑ |
159
+ |---|---:|---:|---:|---:|---:|---:|---:|---:|---:|
160
+ | BF16 reference | 7.755 | 9.580 | 8.668 | 0.64703 | 0.89293 | 0.76998 | 91.000 | 0.90314 | 0.85792 |
161
+ | H64 mixed NVFP4/BF16 | 7.938 | 9.672 | 8.805 | 0.64536 | 0.89038 | 0.76787 | 91.000 | 0.91393 | 0.88300 |
162
+
163
+ #### Paired differences and confidence intervals
164
+
165
+ Differences are H64 minus BF16. WER and ECA differences use percentage points; similarity metrics are dimensionless. The 95% intervals use 10,000 paired bootstrap resamples by the 30 parallel ESD sentence positions within each speaker group, conditional on the fixed speakers. They cover the five aggregate metrics below; no intervals for the individual ASR or speaker evaluators are inferred.
166
+
167
+ | Speaker group | Metric | Difference | Paired 95% CI |
168
+ |---|---|---:|---|
169
+ | Seen | WER avg (pp) | -0.074 | [-0.288, 0.162] |
170
+ | Seen | SECS avg | -0.002357 | [-0.002870, -0.001841] |
171
+ | Seen | ECA (pp) | +0.167 | [-0.583, 0.917] |
172
+ | Seen | EECS | +0.003514 | [0.000334, 0.006857] |
173
+ | Seen | SVAS | +0.003741 | [-0.004099, 0.012922] |
174
+ | Unseen | WER avg (pp) | +0.137 | [-0.330, 0.567] |
175
+ | Unseen | SECS avg | -0.002109 | [-0.003227, -0.001027] |
176
+ | Unseen | ECA (pp) | +0.000 | [-1.333, 1.333] |
177
+ | Unseen | EECS | +0.010793 | [0.000101, 0.020127] |
178
+ | Unseen | SVAS | +0.025080 | [0.006190, 0.049711] |
179
+
180
+ H64 changes WER avg by -0.074 percentage points on seen speakers and +0.137 on unseen speakers. ECA changes by +0.167 points on seen speakers and zero on unseen speakers. SECS avg decreases by 0.002357 and 0.002109 respectively, with negative paired intervals in both groups. EECS and SVAS point estimates increase. These are measured tradeoffs of the optional H64 variant on this benchmark; they do not establish a population-level equivalence margin or a uniform quality improvement.
181
+
182
+ Exact utterance overlap between the 240-recording selection and the full benchmark is zero for selection-target/benchmark-target, selection-target/benchmark-reference, selection-reference/benchmark-target and selection-reference/benchmark-reference comparisons. However, all 1,500 benchmark target recordings were evaluated with earlier FP32/NVFP4 configurations during the historical search. The full benchmark is therefore not an untouched holdout. Its paired intervals are conditional on these fixed speakers and sentence positions; they do not account for adaptive selection or generalize uncertainty to arbitrary speakers.
183
+
184
+ The BF16 estimator timestep input is part of the matched runtime contract. The FP32 scalar integration schedule is retained as solver metadata, without an FP32 inference-weight copy. No controlled inference-speed comparison is reported.
185
+
186
+ ### Export verification
187
 
188
+ A fresh `python -I` process loaded only `AutoProcessor` and `AutoModel` from this repository at code/weight revision `39f692e1ca0fc42b0d29589c946aa3aaed265f39`, with no project-package import or generation hooks. For ESD seen clip `0014_001095` and unseen clip `0013_001433`, processor tensors, generated mels and postprocessed waveforms exactly matched the saved strict BF16/H64 reference pipeline, including the lazy internal BF16 BigVGAN. Verification seeds 777700 and 777701 used stride-preserving `randn_like` noise; this two-clip parity check uses different noise from the full benchmark and establishes export parity, not quality equivalence. Packed state matched the frozen benchmark before and after generation. No optimizer step was performed.
189
 
190
  ### Paper context
191
 
192
+ The following published EmoSphere++ results are from Table II (seen speakers) and Table III (unseen speakers) of [Cho et al.](https://ieeexplore.ieee.org/document/10965917), using the paper's ESD protocol, vocoder and evaluation setup. They provide source context separately from this conversion's matched BF16/H64 results. Different vocoder weights and evaluation details prevent treating the paper rows as matched comparisons to this conversion.
193
 
194
  | Published model and speaker group | WER avg (%) ↓ | SECS avg ↑ | ECA (%) ↑ | SVAS ↑ | EECS ↑ |
195
  |---|---:|---:|---:|---:|---:|
 
211
  ## Limitations
212
 
213
  - English only, trained on 10 ESD speakers. The selection rows cover ESD recording conditions; they do not establish quality on arbitrary voices or other recording conditions.
214
+ - The full ESD comparison is conditional on a historically inspected benchmark and fixed speakers. No formal population-level quality equivalence or inference speedup is claimed.
215
  - The spherical vector is computed from the emotion label and the reference recording; an emotion other than the five ESD emotions is not supported.
216
  - The text front end rejects numeric digits. Write numbers as English words before calling the processor.
217
  - The vocoder is ours, not the authors'; results differ from the paper by that and by evaluation details.