owensong commited on
Commit
fa4ccd2
·
verified ·
1 Parent(s): 8a43314

Stage verified Inflect v2 release candidate and evaluation evidence

Browse files
Files changed (2) hide show
  1. docs/EVALUATION.md +4 -4
  2. release_manifest.json +2 -2
docs/EVALUATION.md CHANGED
@@ -2,20 +2,20 @@
2
 
3
  ## Matched corpus
4
 
5
- All automated headline results use 500 identical unseen English prompts per system. The set combines 200 manually audited stress prompts with 300 deterministic CMU Arctic sentences. Exact normalized text was excluded against 87,362 training transcripts. The frozen prompt manifest and exclusion report are included under `evaluation/final/raw/`.
6
 
7
  ## Systems
8
 
9
  - Inflect-Micro-v2 and Inflect-Nano-v2, fixed release checkpoints.
10
  - KittenTTS Nano, Bruno and Hugo reported separately.
11
  - Piper low, Ryan and Danny reported separately.
12
- - Supertonic 3 James at 3 and 8 steps, reported separately.
13
 
14
- Family averages for Kitten and Piper are equal-weight macro averages of the two named voices. The two Supertonic step counts are never pooled.
15
 
16
  ## Intelligibility
17
 
18
- Headline semantic WER uses Whisper-large-v3. A second complete pass with wav2vec2 Large LV-60K tests whether conclusions are sensitive to one ASR family. References and hypotheses are normalized into equivalent spoken English forms so formatting differences such as `12.6` versus `twelve point six` are not automatically counted as speech errors. Both reports include per-clip rows, prompt-category breakdowns, and 10,000-sample bootstrap intervals. ASR-family results are never averaged into a single headline value.
19
 
20
  ## Predicted quality
21
 
 
2
 
3
  ## Matched corpus
4
 
5
+ Modern400 intelligibility results use 400 identical unseen English prompts per system. The set combines 200 fixed modern/stress prompts with 200 deterministic Google FLEURS `en_us` test prompts. Exact normalized text was excluded against 87,362 training transcripts. The frozen prompt manifest, corpus hash, exclusion report, compact summary, and compressed row-level reports are included under `evaluation/final/modern400/`. UTMOS22 remains a separate matched quality evaluation and is not merged with Modern400.
6
 
7
  ## Systems
8
 
9
  - Inflect-Micro-v2 and Inflect-Nano-v2, fixed release checkpoints.
10
  - KittenTTS Nano, Bruno and Hugo reported separately.
11
  - Piper low, Ryan and Danny reported separately.
12
+ - Supertonic 3 v1.3.1, official voice M2 at 3 and 8 steps, reported separately.
13
 
14
+ Kitten and Piper voices are reported separately. The two Supertonic step counts are never pooled.
15
 
16
  ## Intelligibility
17
 
18
+ Semantic WER is run independently with Qwen3-ASR-1.7B and Nemotron 3.5 ASR 0.6B. References and hypotheses are normalized into equivalent spoken English forms so formatting differences such as `12.6` versus `twelve point six` are not automatically counted as speech errors. Both reports include every hypothesis, per-clip errors, prompt-category breakdowns, and 10,000-sample bootstrap intervals. ASR-family results are never averaged into a single headline value.
19
 
20
  ## Predicted quality
21
 
release_manifest.json CHANGED
@@ -89,8 +89,8 @@
89
  },
90
  {
91
  "path": "docs/EVALUATION.md",
92
- "bytes": 2401,
93
- "sha256": "fcf1450317c47bc719c3c34f5109286faf81e8ab958073b345431054a2b376bc"
94
  },
95
  {
96
  "path": "docs/FINETUNING.md",
 
89
  },
90
  {
91
  "path": "docs/EVALUATION.md",
92
+ "bytes": 2487,
93
+ "sha256": "05d2dcf13660693890ed329ce3dab3653a5ee6f96b1fc75a15dd0a2c9b031f93"
94
  },
95
  {
96
  "path": "docs/FINETUNING.md",