mesm-tts: ten tiny English voices (ONNX + pdn WebAssembly engine), inference, samples, benchmarks
931612f | # Benchmarks: data and methodology | |
| The main card ([`../README.md`](../README.md)) summarises these. This page has the full tables, the raw data files and the exact protocols. | |
| | file | what | | |
| |---|---| | |
| | `speed_native_onnx.json` | native onnxruntime speed of every ONNX file in this repo, measured 2026-11-05 (protocol inside) | | |
| | `browser_ladder_heart.json` | Heart's progression ladder, v1.1 to v2g: native / onnxruntime-web / pdn speed, size, parameters, first audio, memory; phone proxies; WebGPU; runtime download sizes (the data behind the mesmer.tools TTS Generator's speed story) | | |
| | `../voices.json` | per voice: files (bytes, sha256), parameters, level, joins, and the quality numbers below | | |
| ## 1. Native speed (onnxruntime, 1 thread, reserved core) | |
| - Runtime: onnxruntime 1.31.0, CPUExecutionProvider, intra_op_num_threads=1, inter_op_num_threads=1. | |
| - CPU: one reserved Intel Core i7-14700K P-core (clock locked at 4.5 GHz; other processes, IRQs and kernel work kept off it). | |
| - Input: phoneme ids of the 200 TEST sentences (speed = 1.0); synthesis only (no G2P). | |
| - Schedule: 3 rounds; every model runs sentence i back to back before any moves to i+1, order rotated per sentence; 2 ORT sessions per model alternated; warm-up first. | |
| - Metrics: xRT = total audio seconds / total wall seconds per round, median of rounds; vs_v1_1 = median over rounds of xRT / the v1.1 baseline's xRT in the same round; vs_v1_1_pooled = the ratio pooled over all (sentence, round) pairs, with a bootstrap 95% CI. | |
| | model | file | x real time (median) | per round | ms per audio second | vs v1.1 (median of rounds) | vs v1.1 pooled [95% CI] | | |
| |---|---|---:|---|---:|---:|---:| | |
| | Paradee-8M v1.1 int8 (baseline) | `sahilmahendrakar/Paradee-8M-v1.1: onnx/paradee_int8.onnx` | 42.3 | 37.4 / 42.3 / 43.5 | 23.62 | 1.000 | 1.000 [1.000, 1.000] | | |
| | heart_v2f_lite_noint8 | `onnx/heart/v2f_lite_noint8.onnx` | 224.1 | 204.7 / 224.1 / 232.0 | 4.46 | 5.328 | 5.366 [5.316, 5.411] | | |
| | heart_v2f_lite | `onnx/heart/v2f_lite.onnx` | 273.8 | 244.0 / 273.8 / 281.7 | 3.65 | 6.470 | 6.486 [6.415, 6.557] | | |
| | heart_v2f_noint8 | `onnx/heart/v2f_noint8.onnx` | 197.4 | 181.6 / 197.4 / 203.0 | 5.07 | 4.663 | 4.729 [4.689, 4.770] | | |
| | heart_v2f | `onnx/heart/v2f.onnx` | 234.1 | 208.9 / 234.1 / 238.9 | 4.27 | 5.529 | 5.534 [5.477, 5.591] | | |
| | heart_v2g_noint8 | `onnx/heart/v2g_noint8.onnx` | 237.8 | 215.7 / 237.8 / 242.0 | 4.21 | 5.617 | 5.650 [5.584, 5.711] | | |
| | heart_v2g | `onnx/heart/v2g.onnx` | 291.5 | 262.4 / 291.5 / 300.1 | 3.43 | 6.892 | 6.931 [6.858, 7.002] | | |
| | bryony | `onnx/voices/bryony.onnx` | 105.5 | 97.9 / 105.5 / 108.0 | 9.48 | 2.491 | 2.532 [2.509, 2.556] | | |
| | edwin | `onnx/voices/edwin.onnx` | 82.0 | 76.3 / 82.0 / 84.0 | 12.19 | 1.938 | 1.971 [1.953, 1.988] | | |
| | everly | `onnx/voices/everly.onnx` | 105.7 | 98.0 / 105.7 / 108.6 | 9.46 | 2.496 | 2.539 [2.515, 2.563] | | |
| | hadley | `onnx/voices/hadley.onnx` | 104.1 | 95.5 / 104.1 / 105.8 | 9.61 | 2.458 | 2.482 [2.459, 2.505] | | |
| | hugh | `onnx/voices/hugh.onnx` | 105.8 | 97.6 / 105.8 / 108.3 | 9.45 | 2.499 | 2.534 [2.513, 2.556] | | |
| | maeve | `onnx/voices/maeve.onnx` | 105.1 | 96.1 / 105.1 / 106.9 | 9.51 | 2.483 | 2.504 [2.478, 2.529] | | |
| | miles | `onnx/voices/miles.onnx` | 105.0 | 97.9 / 105.0 / 107.9 | 9.53 | 2.479 | 2.526 [2.507, 2.546] | | |
| | trevor | `onnx/voices/trevor.onnx` | 105.6 | 97.0 / 105.6 / 108.3 | 9.47 | 2.493 | 2.526 [2.509, 2.545] | | |
| | wren | `onnx/voices/wren.onnx` | 105.1 | 96.4 / 105.1 / 108.1 | 9.52 | 2.483 | 2.515 [2.499, 2.533] | | |
| Output lengths were identical across rounds for every model (`length_varies` false): durations do not depend on the noise draw. | |
| ## 2. Browser: the Heart ladder (one run per runtime, 1 thread, reserved core, V8 / Node 26, TEST-200) | |
| `web` = onnxruntime-web 1.22 WASM (the build Chickadee ships), `pdn` = our engine with the relaxed-SIMD build and int8 math, `pdn128` = the plain SIMD128 build. `ttfa` = a fresh process to the first audio of a short sentence (engine compile + weights + synthesis, download excluded). `rss` = peak memory of the whole process (Node itself is about 40 MB). | |
| | step | MB | params (M) | native | web | pdn | pdn128 | ttfa web / pdn (ms) | rss web / pdn (MB) | | |
| |---|---:|---:|---:|---:|---:|---:|---:|---:| | |
| | v1.1 | 8.46 | 7.57 | 39.0x | 15.1x | | | 3558 / | 646 / | | |
| | v1.1_fast | 8.12 | 7.57 | 65.7x | 28.3x | | | 2993 / | 708 / | | |
| | v2a | 4.60 | 5.99 | 71.3x | 32.5x | | | 3058 / | 918 / | | |
| | v2a_sep | 4.61 | 5.99 | 82.5x | 34.2x | | | 3305 / | 901 / | | |
| | v2d | 5.00 | 5.94 | 102.0x | 40.8x | 81.6x | 68.7x | 3412 / 112 | 928 / 143 | | |
| | v2e_base | 4.26 | 4.94 | 110.2x | 45.1x | 94.6x | 80.0x | 3080 / 100 | 845 / 133 | | |
| | v2e_noint8 | 3.98 | 4.94 | 120.4x | 47.1x | 94.7x | 79.9x | 4144 / 100 | 1717 / 131 | | |
| | v2e | 4.01 | 4.94 | 139.4x | 50.4x | 120.0x | 87.9x | 4180 / 100 | 1820 / 131 | | |
| | v2f_r8b | 3.56 | 4.17 | 143.6x | 54.0x | 128.0x | 95.1x | 4238 / 90 | 1879 / 121 | | |
| | v2f_av4_fp | 3.40 | 4.10 | 184.4x | 68.0x | 123.2x | 102.7x | 2370 / 84 | 796 / 117 | | |
| | v2f_av4 | 3.42 | 4.10 | 213.0x | 73.4x | 148.7x | 108.0x | 2644 / 84 | 849 / 117 | | |
| | v2f_t1v4_fp | 2.61 | 3.25 | 209.2x | 78.2x | 147.4x | 123.6x | 2269 / 75 | 768 / 106 | | |
| | v2f_t1v4 | 2.64 | 3.25 | 252.8x | 84.8x | 185.2x | 131.2x | 2647 / 75 | 774 / 106 | | |
| | v2g_fp | 2.29 | 2.91 | 245.4x | 87.0x | 156.9x | 131.7x | 2069 / 60 | 757 / 100 | | |
| | v2g | 2.32 | 2.91 | 304.1x | 96.5x | 201.3x | 140.3x | 2285 / 60 | 770 / 100 | | |
| Step names: `v2f_av4` = Heart v2f, `v2f_t1v4` = Heart v2f lite, `*_fp` / `*_noint8` = the same weights with fp32 math. | |
| **Download (runtime only):** ort-web 1.22 jsep wasm (Chickadee): 21.596 MB (2.988 MB brotli); ort-web 1.22 js (ort.min.mjs): 0.355 MB (0.079 MB brotli); pdn wasm SIMD128: 0.136 MB (0.038 MB brotli); pdn wasm relaxed-SIMD: 0.136 MB (0.038 MB brotli); pdn js (pdn.mjs): 0.007 MB (0.003 MB brotli). | |
| **WebGPU** (onnxruntime-web, Heart v2f fp32 twin, RTX 4070 Ti SUPER, headless Chrome, 20 TEST sentences x 2 rounds): 112.8x real time after 1753 ms of start-up, against onnxruntime-web WASM 1 thread 89.1x and pdn 1 thread 184.1x (83 ms cold start). It cannot stream (one session run per sentence). | |
| **Phone proxies** (the same benchmark on a desktop core slowed by the factor shown; streaming first audio for pdn, whole sentence for onnxruntime-web): | |
| | slowdown | model / runtime | x real time | vs v1.1 | first audio (ms) | | |
| |---|---|---:|---:|---:| | |
| | 1.0x | v11_ort | 15.7 | 1 | 465 | | |
| | 1.0x | v2f_ort | 77.9 | 4.97 | 96 | | |
| | 1.0x | v2g_ort | 94.3 | 6.03 | 79 | | |
| | 1.0x | v2f_pdn | 144.5 | 9.23 | 32 | | |
| | 1.0x | v2f_pdn128 | 107.1 | 6.84 | 41 | | |
| | 1.0x | v2g_pdn | 194.4 | 12.42 | 11 | | |
| | 1.0x | v2g_pdn128 | 138.0 | 8.82 | 13 | | |
| | 1.0x | lite_ort | 89.9 | 5.75 | 83 | | |
| | 1.0x | lite_pdn | 184.6 | 11.79 | 22 | | |
| | 1.0x | lite_pdn128 | 130.6 | 8.35 | 29 | | |
| | 3.84x | v11_ort | 3.9 | 1 | 1838 | | |
| | 3.84x | v2f_ort | 19.2 | 4.88 | 384 | | |
| | 3.84x | v2g_ort | 23.1 | 5.88 | 308 | | |
| | 3.84x | v2f_pdn | 36.9 | 9.38 | 127 | | |
| | 3.84x | v2f_pdn128 | 27.1 | 6.89 | 162 | | |
| | 3.84x | v2g_pdn | 49.3 | 12.55 | 45 | | |
| | 3.84x | v2g_pdn128 | 34.7 | 8.83 | 55 | | |
| | 3.84x | lite_ort | 22.4 | 5.69 | 326 | | |
| | 3.84x | lite_pdn | 46.0 | 11.7 | 90 | | |
| | 3.84x | lite_pdn128 | 32.8 | 8.35 | 113 | | |
| | 5.7x | v11_ort | 2.6 | 1 | 2829 | | |
| | 5.7x | v2f_ort | 12.4 | 4.84 | 590 | | |
| | 5.7x | v2g_ort | 15.1 | 5.88 | 494 | | |
| | 5.7x | v2f_pdn | 24.1 | 9.4 | 202 | | |
| | 5.7x | v2f_pdn128 | 17.6 | 6.85 | 251 | | |
| | 5.7x | v2g_pdn | 32.3 | 12.58 | 67 | | |
| | 5.7x | v2g_pdn128 | 22.7 | 8.83 | 79 | | |
| | 5.7x | lite_ort | 14.6 | 5.69 | 512 | | |
| | 5.7x | lite_pdn | 30.3 | 11.81 | 137 | | |
| | 5.7x | lite_pdn128 | 21.4 | 8.35 | 181 | | |
| **iPhone 13 Pro** (Safari, SIMD128 build; the live mesmer.tools TTS Generator's own counter, measured by hand, not under this protocol): Heart about 50x real time, the Phonon voices about 33x. | |
| ## 3. Browser: the Phonon voices (pdn, fp32 math, 1 thread, reserved core, TEST-200, interleaved with Heart v2f) | |
| | voices | SIMD128 | relaxed SIMD | streaming first audio, median / p90 (relaxed) | cold start to first audio | peak RSS | | |
| |---|---:|---:|---:|---:|---:| | |
| | Wren, Everly, Hadley, Bryony, Maeve, Miles, Trevor, Hugh (head 160 x 4) | 66.6x | 80.0x | 54.9 / 84.8 ms | 111 to 115 ms | 140 MB | | |
| | Edwin (head 256 x 6) | 45.5x | 54.7x | | | | | |
| int8 math is off for these voices: their decoders were not trained for it (it cost 0.02 UTMOS on DEV). | |
| ## 4. Quality: Heart on TEST-200 (held out; 3 seeds, v1.1 5 seeds) | |
| | | Paradee-8M v1.1 | v2f | v2f lite | v2g | Kokoro-82M | | |
| |---|---:|---:|---:|---:|---:| | |
| | UTMOS | 4.391 | 4.462 | 4.453 | 4.453 | 4.52 | | |
| | WER Whisper-base | 5.85% | 5.65% | 5.64% | 5.48% | 5.67% | | |
| | WER capped (3 / sentence) | 5.17% | 4.91% | 4.84% | 4.91% | | | |
| | WER large-v3-turbo | 3.07% | 3.37% | | 3.21% | 3.35% | | |
| Paired comparisons (per-sentence bootstrap, 95%): v2f vs v1.1 UTMOS +0.071 [+0.058, +0.084] (significant), WER −0.19 pp [−0.64, +0.22] (tie); v2f lite vs v1.1 UTMOS +0.062 (significant), WER −0.21 pp (tie); v2g vs v1.1 UTMOS +0.062 (significant), WER −0.36 pp (tie), turbo WER +0.19 pp [−0.14, +0.56] (tie). v2f lite and v2g vs v2f: UTMOS −0.009 [−0.021, +0.003] and −0.009 [−0.019, +0.001] (ties). | |
| ## 5. Quality: the Phonon voices on DEV-400 (student 3 seeds vs its own teacher 2 seeds) | |
| | voice | UTMOS student / teacher | gap [95% CI] | WER student / teacher | capped WER student / teacher | capped diff [95% CI] | turbo WER student / teacher | speaker sim. (ceiling) | | |
| |---|---:|---:|---:|---:|---:|---:|---:| | |
| | Wren | 4.417 / 4.452 | +0.035 [+0.024, +0.047] | 6.24% / 6.11% | 5.66% / 5.50% | [-0.26, +0.61] pp | 3.48% / 3.61% | 0.857 (0.923) | | |
| | Everly | 4.346 / 4.434 | +0.088 [+0.076, +0.100] | 6.09% / 6.12% | 5.34% / 5.47% | [-0.54, +0.29] pp | 3.40% / 3.91% | 0.877 (0.935) | | |
| | Hadley | 4.469 / 4.496 | +0.026 [+0.019, +0.035] | 5.98% / 6.10% | 5.52% / 5.47% | [-0.41, +0.51] pp | 3.21% / 3.56% | 0.882 (0.937) | | |
| | Bryony | 4.172 / 4.331 | +0.159 [+0.145, +0.173] | 6.95% / 7.43% | 6.22% / 6.52% | [-0.78, +0.21] pp | 3.56% / 4.17% | 0.856 (0.934) | | |
| | Maeve | 4.391 / 4.442 | +0.052 [+0.042, +0.061] | 6.31% / 6.20% | 5.75% / 5.64% | [-0.39, +0.59] pp | 3.35% / 3.86% | 0.880 (0.940) | | |
| | Miles | 4.258 / 4.365 | +0.107 [+0.092, +0.122] | 6.17% / 6.45% | 5.46% / 5.53% | [-0.49, +0.36] pp | 3.26% / 3.80% | 0.880 (0.933) | | |
| | Trevor | 4.323 / 4.425 | +0.102 [+0.090, +0.114] | 6.23% / 6.17% | 5.55% / 5.42% | [-0.30, +0.55] pp | 3.07% / 3.61% | 0.855 (0.941) | | |
| | Edwin | 4.216 / 4.389 | +0.173 [+0.153, +0.193] | 6.05% / 6.16% | 5.44% / 5.55% | [-0.56, +0.34] pp | 3.06% / 3.60% | 0.870 (0.938) | | |
| | Hugh | 4.267 / 4.367 | +0.100 [+0.089, +0.111] | 6.83% / 6.58% | 5.94% / 5.78% | [-0.35, +0.64] pp | 3.46% / 3.99% | 0.880 (0.949) | | |
| Teachers were rendered with Phonon-31m-en at temperature 0.3 (the corpus setting). Students are the trained checkpoints in fp32; the shipped packed files match them (parity gate on all 400 sentences: durations identical, network stages within 5e-6, F0 within 0.01 Hz, F0-locked waveform within 1e-4), and spot re-scores of shipped files on DEV were within noise (Wren: UTMOS +0.003 [−0.005, +0.011], WER +0.32 pp [−0.27, +1.05]; Hadley: UTMOS −0.002 [−0.009, +0.005], WER −0.14 pp [−0.44, +0.20]). | |
| ## 6. The evaluation sets and metrics | |
| - **TEST-200**: the first 200 rows of upstream Paradee's `wikitext_sents.json` ([github.com/sahilmahendrakar/paradee](https://github.com/sahilmahendrakar/paradee)), the held-out set of Paradee-8M v1.1's card. Final confirmation only, once per release. | |
| - **DEV-400**: 400 WikiText-103 sentences extracted with upstream's recipe, but only from articles that none of the training sentences come from. Every selection decision. | |
| - **XDEV-1600**: 1,600 held-out sentences from the training pool's sources, without digits (27k words), for WER differences of 0.1 pp. | |
| - **Training text**: upstream's 12k WikiText-103 sentences minus TEST, plus an extra corpus of about 62k sentences from several sources (about 143 h of Kokoro renders for Heart; for each Phonon voice, a corpus of that voice's renders). | |
| - **WER**: OpenAI Whisper `base`, `language="en"`, fp16 off, CPU; reference and hypothesis lowercased with non-alphanumerics removed (upstream's normalisation). Capped WER counts at most 3 errors per sentence, so one truncated or letter-spelled transcript cannot decide a 0.1 pp comparison. Guard: Whisper large-v3-turbo, raw and capped. | |
| - **UTMOS**: UTMOS22-strong (`tarepan/SpeechMOS` v1.2.0). | |
| - **Speaker similarity**: cosine similarity of speaker-verification embeddings, student vs its teacher on the same sentence; the ceiling is the teacher against its own second render. | |
| - **Noise rule**: a difference counts only if its paired bootstrap 95% interval excludes 0 and it exceeds the measured seed noise (5 seeds: about 0.005 UTMOS and 0.1 pp WER on 200 sentences; DEV-400 x3: 0.008 UTMOS and 0.15 pp WER). | |