Model card: report the sealed speakers with listener-checked labels; drop the held-out set

README.md

@@ -73,41 +73,45 @@ The readouts are then trained on the frozen trunk. Each uses importance weights
 
 ## Evaluation
 
-All numbers come from scoring the audio end to end with the released checkpoint.
+All numbers come from scoring the audio end to end with the released checkpoint. Real speech is a sealed set of Common Voice speakers who appear in no training, distillation or development data, and who were used only for reporting, never to choose a design or a setting.
 
 | | Nano | Tiny | Mini | Small |
 |---|---:|---:|---:|---:|
-| Length-matched AUC, held-out Common Voice speakers (2,611 Taiwan, 2,616 Mainland clips) | 0.939 | 0.950 | 0.952 | 0.955 |
-| The same, with the text | 0.938 | 0.949 | 0.951 | 0.954 |
-| Length-matched AUC, a further set of new speakers used for no decision (552 Taiwan, 1,568 Mainland clips, 462 speakers) | 0.874 | 0.889 | 0.891 | 0.901 |
+| Length-matched AUC, sealed speakers, listener-checked labels (583 Taiwan, 1,451 Mainland clips, 448 speakers) | 0.956 | 0.961 | 0.960 | 0.962 |
+| The same, with the text | 0.956 | 0.960 | 0.960 | 0.962 |
+| The same, Common Voice labels as published (2,060 clips) | 0.888 | 0.903 | 0.905 | 0.913 |
+| Expected calibration error, sealed speakers, listener-checked labels | 0.031 | 0.028 | 0.028 | 0.034 |
 | Same text and voice prompt, BreezyVoice ranked above CosyVoice-300M (1,000 pairs) | 79% | 79% | 80% | 80% |
 | The same contrast in standard deviations of the score on real speech | 0.38 | 0.36 | 0.37 | 0.37 |
+| A comparison of the two synthesisers on 10 random pairs that ranks BreezyVoice first | 99% | 99% | 99% | 99% |
 | Score variance explained by the TTS voice (119 texts × 4 voices of one TTS) | 6.8% | 4.8% | 5.5% | 11.9% |
-| Expected calibration error, held-out speakers | 0.013 | 0.011 | 0.014 | 0.015 |
 | CPU latency for a 5 s clip, audio only, 4 threads / 1 thread | 29 / 74 ms | 51 / 132 ms | 73 / 192 ms | 133 / 355 ms |
 | CPU latency with the text, 4 threads / 1 thread | 35 / 81 ms | 57 / 140 ms | 78 / 200 ms | 139 / 362 ms |
 
-- **Held-out speakers:** never used to train any model.
-- **New speakers:** these appear in no training, development or held-out list, and were used only for reporting, never to choose a design or a setting.
-- **Reference:** on the new-speaker set, an ensemble of three HuBERT-base accent judges (about 400 M parameters in use) reaches 0.908 AUC.
-- **Synthetic contrast:** the 1,000 pairs are 250 held-out Taiwan sentences, each read with four voice prompts.
+- **Listener-checked labels:** a Common Voice label records the language version a volunteer chose, not their accent, and some clips hold no speech or no Mandarin.
+  - One listener heard, blind, the 248 sealed clips that at least half of twelve strong systems got wrong, plus 40 clips that all of them got right.
+  - Clips without speech or not in Mandarin were removed. Clips that leaned the other way were relabelled (100). Clips the listener could not tell keep their label.
+  - On the disputed clips the listener confirmed only one label in five. On the clips all systems got right, they disagreed with a few labels too, so the checked-label AUC is an upper estimate. The comparison between systems is the reliable part.
+- **Reference:** on the same clips, an ensemble of three HuBERT-base accent judges (about 400 M parameters in use, with a forced aligner) reaches 0.964 AUC with listener-checked labels and 0.918 with the published labels.
+- **Synthetic contrast:** the 1,000 pairs are 250 Common Voice zh-TW sentences, each synthesised with four voice prompts.
 - **Voice set:** the four voices come from a single TTS model, so the voice share has a wide interval (8–16% for Small) and may include real differences between the voices.
 - **Latency:** measured on a cloud x86 CPU (Google Colab), batch 1, fp32, median of 20 runs. It includes turning the text into syllables.
 
-**Recording conditions.** Change in length-matched AUC, audio only, on the held-out speakers / the new speakers. MP3, real noise and real rooms were never used in training. Training did use simulated band-limiting and µ-law, which are close to the telephone condition.
+**Recording conditions.** Change in length-matched AUC on the sealed speakers, audio only, listener-checked labels. MP3, real noise and real rooms were never used in training. Training did use simulated band-limiting and µ-law, which are close to the telephone condition.
 
 | Condition | Nano | Tiny | Mini | Small |
 |---|---:|---:|---:|---:|
-| MP3 at 32 kbit/s | −0.002 / −0.005 | −0.003 / −0.006 | −0.002 / −0.006 | −0.002 / −0.002 |
-| Telephone (8 kHz, G.711 µ-law) | −0.017 / −0.024 | −0.009 / −0.026 | −0.006 / −0.031 | −0.008 / −0.027 |
-| Real point-source noise at 10 dB SNR (OpenSLR 28) | −0.043 / −0.065 | −0.034 / −0.054 | −0.027 / −0.055 | −0.025 / −0.037 |
-| Real room reverberation (OpenSLR 28) | −0.014 / −0.051 | −0.018 / −0.052 | −0.014 / −0.047 | −0.014 / −0.045 |
+| MP3 at 32 kbit/s | −0.003 | −0.003 | −0.003 | −0.001 |
+| Telephone (8 kHz, G.711 µ-law) | −0.029 | −0.030 | −0.031 | −0.019 |
+| Real point-source noise at 10 dB SNR (OpenSLR 28) | −0.055 | −0.045 | −0.040 | −0.031 |
+| Real room reverberation (OpenSLR 28) | −0.040 | −0.030 | −0.028 | −0.029 |
 
 ## Limitations
 
 - **The text adds little:** once content is controlled, the text mode is as accurate as the audio mode, not more, at every size.
 - **Southwestern Mandarin:** standard Mandarin from speakers of Sichuan, Chongqing, Guizhou and Yunnan (KeSpeech) is judged Taiwan-accented 20–32% of the time, depending on the size. Standard Mandarin from the other Mainland regions is judged Taiwan-accented 2–8% of the time.
 - **Noise and reverberation:** they lower accuracy, most for the smallest size (see the table above).
+- **Not a language detector:** a clip without speech, or in another language, still gets a score. Check that the audio is Mandarin speech first.
 - **Training domain:** volunteer read speech from Common Voice.
 - **Not evaluated:** Hong Kong, Singapore and Malaysian Mandarin.
 
@@ -117,7 +121,7 @@ Research, and evaluating or steering speech synthesis towards Taiwan Mandarin. I
 
 ## Training data and licences
 
-- **Labelled data:** Common Voice 25 zh-TW and Common Voice 21 zh-CN, both CC0, with speakers disjoint across training, development, held-out and new-speaker sets.
+- **Labelled data:** Common Voice 25 zh-TW and Common Voice 21 zh-CN, both CC0, with speakers disjoint between training, development and the sealed evaluation set.
 - **Distillation audio:** about 106 h, used without labels:
   - 21 h of the Common Voice training audio
   - 60 h of AISHELL-1 and 25 h of THCHS-30, both Apache 2.0