Hugging Face's logo

mesmertech
/
mesm-tts

Text-to-Speech
ONNX
English
mesm-tts
tts
webassembly
wasm
on-device
browser
cpu
tiny
distillation
kokoro
phonon
Model card Files
xet
 Community
mesm-tts / benchmarks /README.md
mesmertech's picture
mesmertech
mesm-tts: ten tiny English voices (ONNX + pdn WebAssembly engine), inference, samples, benchmarks
931612f
preview code
|
Raw History Blame Contribute Delete
12.9 kB

Benchmarks: data and methodology

The main card (../README.md) summarises these. This page has the full tables, the raw data files and the exact protocols.

file what
speed_native_onnx.json native onnxruntime speed of every ONNX file in this repo, measured 2026-11-05 (protocol inside)
browser_ladder_heart.json Heart's progression ladder, v1.1 to v2g: native / onnxruntime-web / pdn speed, size, parameters, first audio, memory; phone proxies; WebGPU; runtime download sizes (the data behind the mesmer.tools TTS Generator's speed story)
../voices.json per voice: files (bytes, sha256), parameters, level, joins, and the quality numbers below

1. Native speed (onnxruntime, 1 thread, reserved core)

  • Runtime: onnxruntime 1.31.0, CPUExecutionProvider, intra_op_num_threads=1, inter_op_num_threads=1.
  • CPU: one reserved Intel Core i7-14700K P-core (clock locked at 4.5 GHz; other processes, IRQs and kernel work kept off it).
  • Input: phoneme ids of the 200 TEST sentences (speed = 1.0); synthesis only (no G2P).
  • Schedule: 3 rounds; every model runs sentence i back to back before any moves to i+1, order rotated per sentence; 2 ORT sessions per model alternated; warm-up first.
  • Metrics: xRT = total audio seconds / total wall seconds per round, median of rounds; vs_v1_1 = median over rounds of xRT / the v1.1 baseline's xRT in the same round; vs_v1_1_pooled = the ratio pooled over all (sentence, round) pairs, with a bootstrap 95% CI.
model file x real time (median) per round ms per audio second vs v1.1 (median of rounds) vs v1.1 pooled [95% CI]
Paradee-8M v1.1 int8 (baseline) sahilmahendrakar/Paradee-8M-v1.1: onnx/paradee_int8.onnx 42.3 37.4 / 42.3 / 43.5 23.62 1.000 1.000 [1.000, 1.000]
heart_v2f_lite_noint8 onnx/heart/v2f_lite_noint8.onnx 224.1 204.7 / 224.1 / 232.0 4.46 5.328 5.366 [5.316, 5.411]
heart_v2f_lite onnx/heart/v2f_lite.onnx 273.8 244.0 / 273.8 / 281.7 3.65 6.470 6.486 [6.415, 6.557]
heart_v2f_noint8 onnx/heart/v2f_noint8.onnx 197.4 181.6 / 197.4 / 203.0 5.07 4.663 4.729 [4.689, 4.770]
heart_v2f onnx/heart/v2f.onnx 234.1 208.9 / 234.1 / 238.9 4.27 5.529 5.534 [5.477, 5.591]
heart_v2g_noint8 onnx/heart/v2g_noint8.onnx 237.8 215.7 / 237.8 / 242.0 4.21 5.617 5.650 [5.584, 5.711]
heart_v2g onnx/heart/v2g.onnx 291.5 262.4 / 291.5 / 300.1 3.43 6.892 6.931 [6.858, 7.002]
bryony onnx/voices/bryony.onnx 105.5 97.9 / 105.5 / 108.0 9.48 2.491 2.532 [2.509, 2.556]
edwin onnx/voices/edwin.onnx 82.0 76.3 / 82.0 / 84.0 12.19 1.938 1.971 [1.953, 1.988]
everly onnx/voices/everly.onnx 105.7 98.0 / 105.7 / 108.6 9.46 2.496 2.539 [2.515, 2.563]
hadley onnx/voices/hadley.onnx 104.1 95.5 / 104.1 / 105.8 9.61 2.458 2.482 [2.459, 2.505]
hugh onnx/voices/hugh.onnx 105.8 97.6 / 105.8 / 108.3 9.45 2.499 2.534 [2.513, 2.556]
maeve onnx/voices/maeve.onnx 105.1 96.1 / 105.1 / 106.9 9.51 2.483 2.504 [2.478, 2.529]
miles onnx/voices/miles.onnx 105.0 97.9 / 105.0 / 107.9 9.53 2.479 2.526 [2.507, 2.546]
trevor onnx/voices/trevor.onnx 105.6 97.0 / 105.6 / 108.3 9.47 2.493 2.526 [2.509, 2.545]
wren onnx/voices/wren.onnx 105.1 96.4 / 105.1 / 108.1 9.52 2.483 2.515 [2.499, 2.533]

Output lengths were identical across rounds for every model (length_varies false): durations do not depend on the noise draw.

2. Browser: the Heart ladder (one run per runtime, 1 thread, reserved core, V8 / Node 26, TEST-200)

web = onnxruntime-web 1.22 WASM (the build Chickadee ships), pdn = our engine with the relaxed-SIMD build and int8 math, pdn128 = the plain SIMD128 build. ttfa = a fresh process to the first audio of a short sentence (engine compile + weights + synthesis, download excluded). rss = peak memory of the whole process (Node itself is about 40 MB).

step MB params (M) native web pdn pdn128 ttfa web / pdn (ms) rss web / pdn (MB)
v1.1 8.46 7.57 39.0x 15.1x 3558 / 646 /
v1.1_fast 8.12 7.57 65.7x 28.3x 2993 / 708 /
v2a 4.60 5.99 71.3x 32.5x 3058 / 918 /
v2a_sep 4.61 5.99 82.5x 34.2x 3305 / 901 /
v2d 5.00 5.94 102.0x 40.8x 81.6x 68.7x 3412 / 112 928 / 143
v2e_base 4.26 4.94 110.2x 45.1x 94.6x 80.0x 3080 / 100 845 / 133
v2e_noint8 3.98 4.94 120.4x 47.1x 94.7x 79.9x 4144 / 100 1717 / 131
v2e 4.01 4.94 139.4x 50.4x 120.0x 87.9x 4180 / 100 1820 / 131
v2f_r8b 3.56 4.17 143.6x 54.0x 128.0x 95.1x 4238 / 90 1879 / 121
v2f_av4_fp 3.40 4.10 184.4x 68.0x 123.2x 102.7x 2370 / 84 796 / 117
v2f_av4 3.42 4.10 213.0x 73.4x 148.7x 108.0x 2644 / 84 849 / 117
v2f_t1v4_fp 2.61 3.25 209.2x 78.2x 147.4x 123.6x 2269 / 75 768 / 106
v2f_t1v4 2.64 3.25 252.8x 84.8x 185.2x 131.2x 2647 / 75 774 / 106
v2g_fp 2.29 2.91 245.4x 87.0x 156.9x 131.7x 2069 / 60 757 / 100
v2g 2.32 2.91 304.1x 96.5x 201.3x 140.3x 2285 / 60 770 / 100

Step names: v2f_av4 = Heart v2f, v2f_t1v4 = Heart v2f lite, *_fp / *_noint8 = the same weights with fp32 math.

Download (runtime only): ort-web 1.22 jsep wasm (Chickadee): 21.596 MB (2.988 MB brotli); ort-web 1.22 js (ort.min.mjs): 0.355 MB (0.079 MB brotli); pdn wasm SIMD128: 0.136 MB (0.038 MB brotli); pdn wasm relaxed-SIMD: 0.136 MB (0.038 MB brotli); pdn js (pdn.mjs): 0.007 MB (0.003 MB brotli).

WebGPU (onnxruntime-web, Heart v2f fp32 twin, RTX 4070 Ti SUPER, headless Chrome, 20 TEST sentences x 2 rounds): 112.8x real time after 1753 ms of start-up, against onnxruntime-web WASM 1 thread 89.1x and pdn 1 thread 184.1x (83 ms cold start). It cannot stream (one session run per sentence).

Phone proxies (the same benchmark on a desktop core slowed by the factor shown; streaming first audio for pdn, whole sentence for onnxruntime-web):

slowdown model / runtime x real time vs v1.1 first audio (ms)
1.0x v11_ort 15.7 1 465
1.0x v2f_ort 77.9 4.97 96
1.0x v2g_ort 94.3 6.03 79
1.0x v2f_pdn 144.5 9.23 32
1.0x v2f_pdn128 107.1 6.84 41
1.0x v2g_pdn 194.4 12.42 11
1.0x v2g_pdn128 138.0 8.82 13
1.0x lite_ort 89.9 5.75 83
1.0x lite_pdn 184.6 11.79 22
1.0x lite_pdn128 130.6 8.35 29
3.84x v11_ort 3.9 1 1838
3.84x v2f_ort 19.2 4.88 384
3.84x v2g_ort 23.1 5.88 308
3.84x v2f_pdn 36.9 9.38 127
3.84x v2f_pdn128 27.1 6.89 162
3.84x v2g_pdn 49.3 12.55 45
3.84x v2g_pdn128 34.7 8.83 55
3.84x lite_ort 22.4 5.69 326
3.84x lite_pdn 46.0 11.7 90
3.84x lite_pdn128 32.8 8.35 113
5.7x v11_ort 2.6 1 2829
5.7x v2f_ort 12.4 4.84 590
5.7x v2g_ort 15.1 5.88 494
5.7x v2f_pdn 24.1 9.4 202
5.7x v2f_pdn128 17.6 6.85 251
5.7x v2g_pdn 32.3 12.58 67
5.7x v2g_pdn128 22.7 8.83 79
5.7x lite_ort 14.6 5.69 512
5.7x lite_pdn 30.3 11.81 137
5.7x lite_pdn128 21.4 8.35 181

iPhone 13 Pro (Safari, SIMD128 build; the live mesmer.tools TTS Generator's own counter, measured by hand, not under this protocol): Heart about 50x real time, the Phonon voices about 33x.

3. Browser: the Phonon voices (pdn, fp32 math, 1 thread, reserved core, TEST-200, interleaved with Heart v2f)

voices SIMD128 relaxed SIMD streaming first audio, median / p90 (relaxed) cold start to first audio peak RSS
Wren, Everly, Hadley, Bryony, Maeve, Miles, Trevor, Hugh (head 160 x 4) 66.6x 80.0x 54.9 / 84.8 ms 111 to 115 ms 140 MB
Edwin (head 256 x 6) 45.5x 54.7x

int8 math is off for these voices: their decoders were not trained for it (it cost 0.02 UTMOS on DEV).

4. Quality: Heart on TEST-200 (held out; 3 seeds, v1.1 5 seeds)

Paradee-8M v1.1 v2f v2f lite v2g Kokoro-82M
UTMOS 4.391 4.462 4.453 4.453 4.52
WER Whisper-base 5.85% 5.65% 5.64% 5.48% 5.67%
WER capped (3 / sentence) 5.17% 4.91% 4.84% 4.91%
WER large-v3-turbo 3.07% 3.37% 3.21% 3.35%

Paired comparisons (per-sentence bootstrap, 95%): v2f vs v1.1 UTMOS +0.071 [+0.058, +0.084] (significant), WER −0.19 pp [−0.64, +0.22] (tie); v2f lite vs v1.1 UTMOS +0.062 (significant), WER −0.21 pp (tie); v2g vs v1.1 UTMOS +0.062 (significant), WER −0.36 pp (tie), turbo WER +0.19 pp [−0.14, +0.56] (tie). v2f lite and v2g vs v2f: UTMOS −0.009 [−0.021, +0.003] and −0.009 [−0.019, +0.001] (ties).

5. Quality: the Phonon voices on DEV-400 (student 3 seeds vs its own teacher 2 seeds)

voice UTMOS student / teacher gap [95% CI] WER student / teacher capped WER student / teacher capped diff [95% CI] turbo WER student / teacher speaker sim. (ceiling)
Wren 4.417 / 4.452 +0.035 [+0.024, +0.047] 6.24% / 6.11% 5.66% / 5.50% [-0.26, +0.61] pp 3.48% / 3.61% 0.857 (0.923)
Everly 4.346 / 4.434 +0.088 [+0.076, +0.100] 6.09% / 6.12% 5.34% / 5.47% [-0.54, +0.29] pp 3.40% / 3.91% 0.877 (0.935)
Hadley 4.469 / 4.496 +0.026 [+0.019, +0.035] 5.98% / 6.10% 5.52% / 5.47% [-0.41, +0.51] pp 3.21% / 3.56% 0.882 (0.937)
Bryony 4.172 / 4.331 +0.159 [+0.145, +0.173] 6.95% / 7.43% 6.22% / 6.52% [-0.78, +0.21] pp 3.56% / 4.17% 0.856 (0.934)
Maeve 4.391 / 4.442 +0.052 [+0.042, +0.061] 6.31% / 6.20% 5.75% / 5.64% [-0.39, +0.59] pp 3.35% / 3.86% 0.880 (0.940)
Miles 4.258 / 4.365 +0.107 [+0.092, +0.122] 6.17% / 6.45% 5.46% / 5.53% [-0.49, +0.36] pp 3.26% / 3.80% 0.880 (0.933)
Trevor 4.323 / 4.425 +0.102 [+0.090, +0.114] 6.23% / 6.17% 5.55% / 5.42% [-0.30, +0.55] pp 3.07% / 3.61% 0.855 (0.941)
Edwin 4.216 / 4.389 +0.173 [+0.153, +0.193] 6.05% / 6.16% 5.44% / 5.55% [-0.56, +0.34] pp 3.06% / 3.60% 0.870 (0.938)
Hugh 4.267 / 4.367 +0.100 [+0.089, +0.111] 6.83% / 6.58% 5.94% / 5.78% [-0.35, +0.64] pp 3.46% / 3.99% 0.880 (0.949)

Teachers were rendered with Phonon-31m-en at temperature 0.3 (the corpus setting). Students are the trained checkpoints in fp32; the shipped packed files match them (parity gate on all 400 sentences: durations identical, network stages within 5e-6, F0 within 0.01 Hz, F0-locked waveform within 1e-4), and spot re-scores of shipped files on DEV were within noise (Wren: UTMOS +0.003 [−0.005, +0.011], WER +0.32 pp [−0.27, +1.05]; Hadley: UTMOS −0.002 [−0.009, +0.005], WER −0.14 pp [−0.44, +0.20]).

6. The evaluation sets and metrics

  • TEST-200: the first 200 rows of upstream Paradee's wikitext_sents.json (github.com/sahilmahendrakar/paradee), the held-out set of Paradee-8M v1.1's card. Final confirmation only, once per release.
  • DEV-400: 400 WikiText-103 sentences extracted with upstream's recipe, but only from articles that none of the training sentences come from. Every selection decision.
  • XDEV-1600: 1,600 held-out sentences from the training pool's sources, without digits (27k words), for WER differences of 0.1 pp.
  • Training text: upstream's 12k WikiText-103 sentences minus TEST, plus an extra corpus of about 62k sentences from several sources (about 143 h of Kokoro renders for Heart; for each Phonon voice, a corpus of that voice's renders).
  • WER: OpenAI Whisper base, language="en", fp16 off, CPU; reference and hypothesis lowercased with non-alphanumerics removed (upstream's normalisation). Capped WER counts at most 3 errors per sentence, so one truncated or letter-spelled transcript cannot decide a 0.1 pp comparison. Guard: Whisper large-v3-turbo, raw and capped.
  • UTMOS: UTMOS22-strong (tarepan/SpeechMOS v1.2.0).
  • Speaker similarity: cosine similarity of speaker-verification embeddings, student vs its teacher on the same sentence; the ceiling is the teacher against its own second render.
  • Noise rule: a difference counts only if its paired bootstrap 95% interval excludes 0 and it exceeds the measured seed noise (5 seeds: about 0.005 UTMOS and 0.1 pp WER on 200 sentences; DEV-400 x3: 0.008 UTMOS and 0.15 pp WER).