| # evaluation protocol |
|
|
| ## indicemo |
|
|
| [repository](https://github.com/ira-rumik/IndicEmo) |
|
|
| 100 prompts span five equally represented delivery categories: happy, sad, |
| angry, excited, and professional. each prompt combines two to five languages |
| from english, hindi, telugu, tamil, kannada, bengali, and punjabi. five systems |
| produce 500 recordings. |
|
|
| gemini 3.1 pro preview, qwen3.5-omni-plus, and gpt-realtime-2.1 score anonymized |
| recordings on a 1-5 rubric covering tone, dynamics, phrasing, and sustained |
| expression. aggregation uses the 98-prompt intersection with valid ratings from |
| all judges: median per recording, mean within category, then an unweighted |
| mean across categories. the emotions-only aggregate excludes professional. |
|
|
| rumik-oss-1 uses ira throughout; competing systems use language-conditioned |
| voices. statistical significance of ranking differences has not been evaluated. |
|
|
| ## nova |
|
|
| [repository](https://github.com/ira-rumik/nova-bench) 路 |
| [protocol](https://github.com/ira-rumik/nova-bench/blob/main/METHODOLOGY.md) |
|
|
| seven systems are evaluated on 149 single-tag english prompts from |
| nvv-superbench, with three synthesis runs per system. prompts use |
| provider-specific tags. rumik-oss-1 uses ira, with happy descriptions for |
| laugh/chuckle and sad descriptions for sigh. |
|
|
| silk-asr, scribe v2, and the nvv-superbench gemini verifier measure exact-position |
| rendering. laugh and chuckle are pooled into a laughter category. the aggregate |
| is the mean of laughter and sigh rendering rates, averaged across detectors. |
| unsupported categories contribute zero, including cartesia's absent sigh tag. |
| detector-level results report means and standard deviations across runs. |
|
|
| unrequested vocalization rates for rumik-oss-1 are 2.9% (silk-asr), 5.4% |
| (gemini), and 4.5% (scribe). gemini is conditioned on the requested event; |
| the transcribers receive audio only. silk-asr is developed by rumik; scribe |
| shares a provider with an evaluated system. |
|
|
| ## wer and cer |
|
|
| [wer results](wer-reported-scores.json) 路 [cer results](cer-reported-scores.json) |
|
|
| the full comparison covers 15 languages. seven-language subsets are selected |
| post hoc by top-two numerical rank among displayed systems. the wer subset |
| includes malayalam in place of telugu. provider coverage varies by language; |
| grok results are available only for hindi. |
|
|
| the wer protocol specifies 100 shared prompts per model and language, scored |
| with indicconformer rnnt using normalized text. the cer evaluation reports |
| 100 rumik recordings per language, except telugu with 99. |
|
|
| competitor sample counts, matched utterance coverage, normalization, and |
| recognizer versions have not been independently verified. confidence intervals |
| and significance tests are unavailable. wer and cer originate from separate |
| result snapshots; matched recordings between them and checkpoint equivalence |
| with the emotion/vocalization evaluations remain unverified. |
|
|
