cloud0day3 commited on
Commit
dab818f
·
verified ·
1 Parent(s): e0d3182

Card: CER lead, WER level (both harness runs)

Browse files
Files changed (1) hide show
  1. README.md +5 -3
README.md CHANGED
@@ -121,9 +121,11 @@ Freya-TR-Eval ([freyavoice/freya-tr-eval](https://huggingface.co/datasets/freyav
121
  | EMA Lightning 1.0.1 (our run) | 8.6M | 1.00 (0.89–1.17) | 0.20 | 3.30 | 2.88 |
122
  | VoxCPM2 speaking the same voice (the teacher; 1 seed) | much larger | 0.92 | 0.20 | 3.42 | 3.25 |
123
 
124
- With its default settings Antalia-2 Mini is ahead of EMA Lightning on WER (0.97% vs 1.00%) and CER
125
- (0.19% vs 0.20%) on our harness, by small margins: both models vary by about ±0.13 WER from seed to
126
- seed. EMA Lightning is ahead on full-band UTMOS (3.30 vs 3.25); Antalia-2 Mini scores higher on UTMOS
 
 
127
  in the 8 kHz band, the band a phone call carries, and is smaller (7.62M vs 8.6M parameters). EMA
128
  Lightning's own card reports 0.92 / 0.18 / ~3.30 on the same protocol.
129
 
 
121
  | EMA Lightning 1.0.1 (our run) | 8.6M | 1.00 (0.89–1.17) | 0.20 | 3.30 | 2.88 |
122
  | VoxCPM2 speaking the same voice (the teacher; 1 seed) | much larger | 0.92 | 0.20 | 3.42 | 3.25 |
123
 
124
+ With its default settings Antalia-2 Mini makes the fewest character errors and is level with EMA
125
+ Lightning on word errors. On this harness: CER 0.19% vs 0.20%, WER 0.97% vs 1.00%. In a second,
126
+ independent run on the public benchmark page's harness (other seeds, one NVIDIA L4): CER 0.20% vs
127
+ 0.22%, the lowest of the 15 Turkish systems measured there, and WER 1.05% vs 1.04%. Both models vary
128
+ by about ±0.1 WER from seed to seed, so the WER difference is within noise either way. EMA Lightning is ahead on full-band UTMOS (3.30 vs 3.25); Antalia-2 Mini scores higher on UTMOS
129
  in the 8 kHz band, the band a phone call carries, and is smaller (7.62M vs 8.6M parameters). EMA
130
  Lightning's own card reports 0.92 / 0.18 / ~3.30 on the same protocol.
131