ArmBench-ASR / index.html
alexshah's picture
added llm scores
fa9b662
Raw History Blame Contribute Delete
64.9 kB
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="description" content="ArmBench-ASR: Benchmarking Speech-to-Text Models on Armenian" />
<title>ArmBench-ASR</title>
<script>
(() => {
document.documentElement.dataset.js = "enabled";
let saved = null;
try {
saved = localStorage.getItem("arm-asr-theme");
} catch (_) {}
const theme = saved || (matchMedia("(prefers-color-scheme: dark)").matches ? "dark" : "light");
document.documentElement.dataset.theme = theme;
})();
</script>
<link rel="preconnect" href="https://fonts.googleapis.com" />
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
<link href="https://fonts.googleapis.com/css2?family=DM+Sans:wght@400;500;600;700&amp;family=Manrope:wght@600;700;800&amp;display=swap" rel="stylesheet" />
<link rel="stylesheet" href="style.css" />
<script src="app.js" defer></script>
</head>
<body>
<main class="shell">
<header class="hero">
<div class="hero-tools"><button class="theme-toggle" id="theme-toggle" type="button" aria-label="Switch color theme"><svg class="sun-icon" viewBox="0 0 24 24" aria-hidden="true">
<circle cx="12" cy="12" r="3.5"></circle>
<path d="M12 2v2m0 16v2M4.9 4.9l1.4 1.4m11.4 11.4 1.4 1.4M2 12h2m16 0h2M4.9 19.1l1.4-1.4M17.7 6.3l1.4-1.4"></path>
</svg><svg class="moon-icon" viewBox="0 0 24 24" aria-hidden="true">
<path d="M20.2 15.1A8.5 8.5 0 0 1 8.9 3.8a8.5 8.5 0 1 0 11.3 11.3Z"></path>
</svg></button></div>
<div class="hero-copy">
<h1><span class="heading-flag" role="img" aria-label="Armenian flag">🇦🇲</span> ArmBench-ASR: Benchmarking Speech-to-Text Models on Armenian</h1>
<p class="lede"><span class="version-badge" id="current-version">v1.0</span></p>
</div>
<div class="hero-art" aria-hidden="true"><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span></div>
</header>
<noscript>
<aside class="noscript-notice" aria-label="JavaScript required for interactive features">
<strong>JavaScript is turned off.</strong> The default leaderboard is available below, but turn on JavaScript to use filters, sorting, charts, and the other interactive views.
</aside>
</noscript>
<div class="navigation-row">
<nav class="tabs" role="tablist" aria-label="Benchmark sections"><button class="tab is-active" role="tab" aria-selected="true" data-tab="leaderboards"> Leaderboards </button><button class="tab" role="tab" aria-selected="false" data-tab="datasets">Datasets</button><button class="tab" role="tab" aria-selected="false" data-tab="about">About</button><button class="tab" role="tab" aria-selected="false" data-tab="changelog">Changelog</button></nav>
<a class="blog-link" href="https://huggingface.co/blog/Metric-AI/armbench-asr" target="_blank" rel="noreferrer">Check out the Blog Post</a>
</div>
<section class="panel is-active matrix-view" id="leaderboards" role="tabpanel">
<div class="table-heading">
<div>
<h2 id="table-title">WER across all datasets</h2>
<p id="table-description"> Rank is based on Overall WER. Scores are percentages; lower is better. </p>
</div>
<div class="table-tools"><span class="result-count" id="result-count" aria-live="polite"><!-- GENERATED:RESULT-COUNT START -->
34 models
<!-- GENERATED:RESULT-COUNT END --></span><label class="version-selector"><span>View versions</span><select id="version-select" aria-label="View benchmark version"><option value="v1.0">v1.0 · Latest</option><option value="v0.1">v0.1</option></select></label></div>
</div>
<aside class="version-notice" id="old-version-notice" hidden>
<strong>You’re viewing an older benchmark version.</strong>
<span>For the latest methodology and results, please view v1.0.</span>
<button class="version-link" id="new-version-link" type="button">View latest results</button>
</aside>
<div class="table-controls" aria-label="Leaderboard display and filters">
<div class="view-switch" role="group" aria-label="Leaderboard view"><button class="view-button is-active" type="button" data-view="matrix" aria-pressed="true"> All datasets </button><button class="view-button" type="button" data-view="single" aria-pressed="false"> Single dataset </button></div>
<div class="filter-card">
<fieldset class="field dataset-field">
<legend>Dataset</legend>
<div class="choice-options" id="dataset-options"></div>
</fieldset>
<fieldset class="field matrix-metric-field">
<legend>Metric across all datasets</legend>
<div class="choice-options" id="matrix-metric-options"></div>
</fieldset>
<div class="filter-actions"><button class="color-toggle" id="color-toggle" type="button" aria-pressed="false"><span class="color-swatch good"></span><span class="color-swatch mid"></span><span class="color-swatch bad"></span> Colors </button></div>
</div>
</div>
<div class="table-card">
<div class="table-scroll">
<table id="leaderboard-table">
<!-- GENERATED:LEADERBOARD START -->
<thead><tr><th title="Ranked by WER for Combined">Rank</th><th>Model</th><th>Availability</th><th class="combined-score">Combined</th><th>Common Voice 26</th><th>FLEURS</th><th>Poems</th><th>Movies</th><th>News</th></tr></thead>
<tbody>
<tr><td class="rank rank-top">#1</td><td><div class="model-cell"><span class="model-name">google/gemini-2.5-pro</span><a class="model-link" href="https://ai.google.dev/gemini-api/docs/models/gemini-2.5-pro" target="_blank" rel="noreferrer" title="Open google/gemini-2.5-pro" aria-label="Open google/gemini-2.5-pro model page"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M14 5h5v5m0-5-8 8M19 14v4a1 1 0 0 1-1 1H6a1 1 0 0 1-1-1V6a1 1 0 0 1 1-1h4"/></svg></a></div></td><td><span class="availability-badge closed">Closed</span></td><td class="combined-score">13.80</td><td>8.92</td><td>7.91</td><td>20.08</td><td>39.19</td><td>11.26</td></tr>
<tr><td class="rank rank-top">#2</td><td><div class="model-cell"><span class="model-name">hispeech/model-23012026</span><span class="model-origin-flag" title="Armenian-origin model" aria-label="Armenian-origin model"></span><a class="model-link" href="https://www.hispeech.ai/api.html" target="_blank" rel="noreferrer" title="Open hispeech/model-23012026" aria-label="Open hispeech/model-23012026 model page"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M14 5h5v5m0-5-8 8M19 14v4a1 1 0 0 1-1 1H6a1 1 0 0 1-1-1V6a1 1 0 0 1 1-1h4"/></svg></a></div></td><td><span class="availability-badge closed">Closed</span></td><td class="combined-score">16.56</td><td>10.94</td><td>13.13</td><td>22.69</td><td>41.35</td><td>14.36</td></tr>
<tr><td class="rank rank-top">#3</td><td><div class="model-cell"><span class="model-name">google/gemini-3-flash-preview</span><a class="model-link" href="https://ai.google.dev/gemini-api/docs/models/gemini-3-flash-preview" target="_blank" rel="noreferrer" title="Open google/gemini-3-flash-preview" aria-label="Open google/gemini-3-flash-preview model page"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M14 5h5v5m0-5-8 8M19 14v4a1 1 0 0 1-1 1H6a1 1 0 0 1-1-1V6a1 1 0 0 1 1-1h4"/></svg></a></div></td><td><span class="availability-badge closed">Closed</span></td><td class="combined-score">16.64</td><td>10.77</td><td>8.99</td><td>24.36</td><td>51.38</td><td>11.06</td></tr>
<tr><td class="rank">#4</td><td><div class="model-cell"><span class="model-name">google/gemini-2.5-flash</span><a class="model-link" href="https://ai.google.dev/gemini-api/docs/models/gemini-2.5-flash" target="_blank" rel="noreferrer" title="Open google/gemini-2.5-flash" aria-label="Open google/gemini-2.5-flash model page"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M14 5h5v5m0-5-8 8M19 14v4a1 1 0 0 1-1 1H6a1 1 0 0 1-1-1V6a1 1 0 0 1 1-1h4"/></svg></a></div></td><td><span class="availability-badge closed">Closed</span></td><td class="combined-score">17.35</td><td>12.23</td><td>10.10</td><td>24.84</td><td>45.87</td><td>13.47</td></tr>
<tr><td class="rank">#5</td><td><div class="model-cell"><span class="model-name">hispeech/model-01052025</span><span class="model-origin-flag" title="Armenian-origin model" aria-label="Armenian-origin model"></span><a class="model-link" href="https://www.hispeech.ai/api.html" target="_blank" rel="noreferrer" title="Open hispeech/model-01052025" aria-label="Open hispeech/model-01052025 model page"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M14 5h5v5m0-5-8 8M19 14v4a1 1 0 0 1-1 1H6a1 1 0 0 1-1-1V6a1 1 0 0 1 1-1h4"/></svg></a></div></td><td><span class="availability-badge closed">Closed</span></td><td class="combined-score">17.63</td><td>10.05</td><td>14.41</td><td>22.96</td><td>49.49</td><td>17.75</td></tr>
<tr><td class="rank">#6</td><td><div class="model-cell"><span class="model-name">talk2edit</span><span class="model-origin-flag" title="Armenian-origin model" aria-label="Armenian-origin model"></span><a class="model-link" href="https://talk2edit.com" target="_blank" rel="noreferrer" title="Open talk2edit" aria-label="Open talk2edit model page"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M14 5h5v5m0-5-8 8M19 14v4a1 1 0 0 1-1 1H6a1 1 0 0 1-1-1V6a1 1 0 0 1 1-1h4"/></svg></a></div></td><td><span class="availability-badge closed">Closed</span></td><td class="combined-score">17.66</td><td>12.30</td><td>10.07</td><td>24.97</td><td>46.11</td><td>15.29</td></tr>
<tr><td class="rank">#7</td><td><div class="model-cell"><span class="model-name">elevenlabs/scribe_v2</span><a class="model-link" href="https://elevenlabs.io/docs/overview/capabilities/speech-to-text" target="_blank" rel="noreferrer" title="Open elevenlabs/scribe_v2" aria-label="Open elevenlabs/scribe_v2 model page"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M14 5h5v5m0-5-8 8M19 14v4a1 1 0 0 1-1 1H6a1 1 0 0 1-1-1V6a1 1 0 0 1 1-1h4"/></svg></a></div></td><td><span class="availability-badge closed">Closed</span></td><td class="combined-score">17.74</td><td>12.10</td><td>9.45</td><td>26.07</td><td>49.82</td><td>13.24</td></tr>
<tr><td class="rank">#8</td><td><div class="model-cell"><span class="model-name">wavam/default</span><span class="model-origin-flag" title="Armenian-origin model" aria-label="Armenian-origin model"></span><a class="model-link" href="https://wav.am" target="_blank" rel="noreferrer" title="Open wavam/default" aria-label="Open wavam/default model page"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M14 5h5v5m0-5-8 8M19 14v4a1 1 0 0 1-1 1H6a1 1 0 0 1-1-1V6a1 1 0 0 1 1-1h4"/></svg></a></div></td><td><span class="availability-badge closed">Closed</span></td><td class="combined-score">19.13</td><td>11.42</td><td>11.91</td><td>30.41</td><td>53.00</td><td>15.67</td></tr>
<tr><td class="rank">#9</td><td><div class="model-cell"><span class="model-name">google/gemini-3.6-flash</span><a class="model-link" href="https://ai.google.dev/gemini-api/docs/models/gemini-3.6-flash" target="_blank" rel="noreferrer" title="Open google/gemini-3.6-flash" aria-label="Open google/gemini-3.6-flash model page"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M14 5h5v5m0-5-8 8M19 14v4a1 1 0 0 1-1 1H6a1 1 0 0 1-1-1V6a1 1 0 0 1 1-1h4"/></svg></a></div></td><td><span class="availability-badge closed">Closed</span></td><td class="combined-score">19.42</td><td>16.04</td><td>11.24</td><td>25.83</td><td>46.64</td><td>12.93</td></tr>
<tr><td class="rank">#10</td><td><div class="model-cell"><span class="model-name">google/gemini-3.5-transcribe-preview</span><a class="model-link" href="https://ai.google.dev/gemini-api/docs/models/gemini-3.5-transcribe" target="_blank" rel="noreferrer" title="Open google/gemini-3.5-transcribe-preview" aria-label="Open google/gemini-3.5-transcribe-preview model page"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M14 5h5v5m0-5-8 8M19 14v4a1 1 0 0 1-1 1H6a1 1 0 0 1-1-1V6a1 1 0 0 1 1-1h4"/></svg></a></div></td><td><span class="availability-badge closed">Closed</span></td><td class="combined-score">19.59</td><td>15.68</td><td>10.95</td><td>28.04</td><td>46.46</td><td>13.26</td></tr>
<tr><td class="rank">#11</td><td><div class="model-cell"><span class="model-name">nvidia/stt_hy_fastconformer_hybrid_large_pc</span><span class="model-origin-flag" title="Armenian-origin model" aria-label="Armenian-origin model"></span><a class="model-link" href="https://huggingface.co/nvidia/stt_hy_fastconformer_hybrid_large_pc" target="_blank" rel="noreferrer" title="Open nvidia/stt_hy_fastconformer_hybrid_large_pc" aria-label="Open nvidia/stt_hy_fastconformer_hybrid_large_pc model page"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M14 5h5v5m0-5-8 8M19 14v4a1 1 0 0 1-1 1H6a1 1 0 0 1-1-1V6a1 1 0 0 1 1-1h4"/></svg></a></div></td><td><span class="availability-badge open">Open</span></td><td class="combined-score">19.86</td><td>7.70</td><td>15.00</td><td>31.28</td><td>60.78</td><td>23.55</td></tr>
<tr><td class="rank">#12</td><td><div class="model-cell"><span class="model-name">wavam/streaming</span><span class="model-origin-flag" title="Armenian-origin model" aria-label="Armenian-origin model"></span><a class="model-link" href="https://wav.am" target="_blank" rel="noreferrer" title="Open wavam/streaming" aria-label="Open wavam/streaming model page"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M14 5h5v5m0-5-8 8M19 14v4a1 1 0 0 1-1 1H6a1 1 0 0 1-1-1V6a1 1 0 0 1 1-1h4"/></svg></a></div></td><td><span class="availability-badge closed">Closed</span></td><td class="combined-score">20.18</td><td>12.29</td><td>14.00</td><td>31.01</td><td>53.75</td><td>16.84</td></tr>
<tr><td class="rank">#13</td><td><div class="model-cell"><span class="model-name">tbb-asr</span><span class="model-origin-flag" title="Armenian-origin model" aria-label="Armenian-origin model"></span><a class="model-link" href="https://asr.tbb.rip" target="_blank" rel="noreferrer" title="Open tbb-asr" aria-label="Open tbb-asr model page"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M14 5h5v5m0-5-8 8M19 14v4a1 1 0 0 1-1 1H6a1 1 0 0 1-1-1V6a1 1 0 0 1 1-1h4"/></svg></a></div></td><td><span class="availability-badge closed">Closed</span></td><td class="combined-score">20.19</td><td>8.15</td><td>15.52</td><td>31.24</td><td>61.67</td><td>23.33</td></tr>
<tr><td class="rank">#14</td><td><div class="model-cell"><span class="model-name">google/gemini-3.1-flash-lite</span><a class="model-link" href="https://ai.google.dev/gemini-api/docs/models/gemini-3.1-flash-lite" target="_blank" rel="noreferrer" title="Open google/gemini-3.1-flash-lite" aria-label="Open google/gemini-3.1-flash-lite model page"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M14 5h5v5m0-5-8 8M19 14v4a1 1 0 0 1-1 1H6a1 1 0 0 1-1-1V6a1 1 0 0 1 1-1h4"/></svg></a></div></td><td><span class="availability-badge closed">Closed</span></td><td class="combined-score">20.27</td><td>16.33</td><td>10.51</td><td>26.08</td><td>54.86</td><td>12.92</td></tr>
<tr><td class="rank">#15</td><td><div class="model-cell"><span class="model-name">openai/gpt-transcribe</span><a class="model-link" href="https://developers.openai.com/api/docs/models/gpt-transcribe" target="_blank" rel="noreferrer" title="Open openai/gpt-transcribe" aria-label="Open openai/gpt-transcribe model page"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M14 5h5v5m0-5-8 8M19 14v4a1 1 0 0 1-1 1H6a1 1 0 0 1-1-1V6a1 1 0 0 1 1-1h4"/></svg></a></div></td><td><span class="availability-badge closed">Closed</span></td><td class="combined-score">22.40</td><td>17.28</td><td>15.37</td><td>30.17</td><td>51.95</td><td>17.24</td></tr>
<tr><td class="rank">#16</td><td><div class="model-cell"><span class="model-name">NCCAIT/model-2024</span><span class="model-origin-flag" title="Armenian-origin model" aria-label="Armenian-origin model"></span><a class="model-link" href="https://media.ican24.net/asr/index.php" target="_blank" rel="noreferrer" title="Open NCCAIT/model-2024" aria-label="Open NCCAIT/model-2024 model page"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M14 5h5v5m0-5-8 8M19 14v4a1 1 0 0 1-1 1H6a1 1 0 0 1-1-1V6a1 1 0 0 1 1-1h4"/></svg></a></div></td><td><span class="availability-badge closed">Closed</span></td><td class="combined-score">25.17</td><td>16.16</td><td>19.49</td><td>33.63</td><td>61.76</td><td>25.83</td></tr>
<tr><td class="rank">#17</td><td><div class="model-cell"><span class="model-name">google/gemini-3.5-flash-lite</span><a class="model-link" href="https://ai.google.dev/gemini-api/docs/models/gemini-3.5-flash-lite" target="_blank" rel="noreferrer" title="Open google/gemini-3.5-flash-lite" aria-label="Open google/gemini-3.5-flash-lite model page"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M14 5h5v5m0-5-8 8M19 14v4a1 1 0 0 1-1 1H6a1 1 0 0 1-1-1V6a1 1 0 0 1 1-1h4"/></svg></a></div></td><td><span class="availability-badge closed">Closed</span></td><td class="combined-score">26.06</td><td>23.69</td><td>14.32</td><td>33.72</td><td>56.01</td><td>16.74</td></tr>
<tr><td class="rank">#18</td><td><div class="model-cell"><span class="model-name">deepgram/nova-3</span><a class="model-link" href="https://developers.deepgram.com/docs/models-languages-overview#nova-3" target="_blank" rel="noreferrer" title="Open deepgram/nova-3" aria-label="Open deepgram/nova-3 model page"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M14 5h5v5m0-5-8 8M19 14v4a1 1 0 0 1-1 1H6a1 1 0 0 1-1-1V6a1 1 0 0 1 1-1h4"/></svg></a></div></td><td><span class="availability-badge closed">Closed</span></td><td class="combined-score">28.46</td><td>7.31</td><td>33.56</td><td>48.35</td><td>75.81</td><td>36.66</td></tr>
<tr><td class="rank">#19</td><td><div class="model-cell"><span class="model-name">facebook/seamless-m4t-v2-large</span><a class="model-link" href="https://huggingface.co/facebook/seamless-m4t-v2-large" target="_blank" rel="noreferrer" title="Open facebook/seamless-m4t-v2-large" aria-label="Open facebook/seamless-m4t-v2-large model page"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M14 5h5v5m0-5-8 8M19 14v4a1 1 0 0 1-1 1H6a1 1 0 0 1-1-1V6a1 1 0 0 1 1-1h4"/></svg></a></div></td><td><span class="availability-badge open">Open</span></td><td class="combined-score">28.94</td><td>15.15</td><td>14.44</td><td>43.93</td><td>80.16</td><td>37.43</td></tr>
<tr><td class="rank">#20</td><td><div class="model-cell"><span class="model-name">chillarmo/whisper-large-v3-turbo-armenian</span><span class="model-origin-flag" title="Armenian-origin model" aria-label="Armenian-origin model"></span><a class="model-link" href="https://huggingface.co/chillarmo/whisper-large-v3-turbo-armenian" target="_blank" rel="noreferrer" title="Open chillarmo/whisper-large-v3-turbo-armenian" aria-label="Open chillarmo/whisper-large-v3-turbo-armenian model page"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M14 5h5v5m0-5-8 8M19 14v4a1 1 0 0 1-1 1H6a1 1 0 0 1-1-1V6a1 1 0 0 1 1-1h4"/></svg></a></div></td><td><span class="availability-badge open">Open</span></td><td class="combined-score">33.00</td><td>12.94</td><td>24.69</td><td>55.09</td><td>82.01</td><td>48.80</td></tr>
<tr><td class="rank">#21</td><td><div class="model-cell"><span class="model-name">arampacha/whisper-large-hy-2</span><span class="model-origin-flag" title="Armenian-origin model" aria-label="Armenian-origin model"></span><a class="model-link" href="https://huggingface.co/arampacha/whisper-large-hy-2" target="_blank" rel="noreferrer" title="Open arampacha/whisper-large-hy-2" aria-label="Open arampacha/whisper-large-hy-2 model page"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M14 5h5v5m0-5-8 8M19 14v4a1 1 0 0 1-1 1H6a1 1 0 0 1-1-1V6a1 1 0 0 1 1-1h4"/></svg></a></div></td><td><span class="availability-badge open">Open</span></td><td class="combined-score">33.17</td><td>23.16</td><td>17.37</td><td>44.48</td><td>77.82</td><td>39.48</td></tr>
<tr><td class="rank">#22</td><td><div class="model-cell"><span class="model-name">modulate-velma-2-stt-batch-multilingual</span><a class="model-link" href="https://docs.modulate.ai/api-reference/stt/batch" target="_blank" rel="noreferrer" title="Open modulate-velma-2-stt-batch-multilingual" aria-label="Open modulate-velma-2-stt-batch-multilingual model page"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M14 5h5v5m0-5-8 8M19 14v4a1 1 0 0 1-1 1H6a1 1 0 0 1-1-1V6a1 1 0 0 1 1-1h4"/></svg></a></div></td><td><span class="availability-badge closed">Closed</span></td><td class="combined-score">33.71</td><td>27.35</td><td>23.95</td><td>37.52</td><td>72.73</td><td>33.77</td></tr>
<tr><td class="rank">#23</td><td><div class="model-cell"><span class="model-name">facebook/omniASR-LLM-3B</span><a class="model-link" href="https://huggingface.co/facebook/omniASR-LLM-3B" target="_blank" rel="noreferrer" title="Open facebook/omniASR-LLM-3B" aria-label="Open facebook/omniASR-LLM-3B model page"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M14 5h5v5m0-5-8 8M19 14v4a1 1 0 0 1-1 1H6a1 1 0 0 1-1-1V6a1 1 0 0 1 1-1h4"/></svg></a></div></td><td><span class="availability-badge open">Open</span></td><td class="combined-score">37.76</td><td>27.45</td><td>26.68</td><td>49.84</td><td>87.43</td><td>35.18</td></tr>
<tr><td class="rank">#24</td><td><div class="model-cell"><span class="model-name">facebook/omniASR-LLM-1B</span><a class="model-link" href="https://huggingface.co/facebook/omniASR-LLM-1B" target="_blank" rel="noreferrer" title="Open facebook/omniASR-LLM-1B" aria-label="Open facebook/omniASR-LLM-1B model page"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M14 5h5v5m0-5-8 8M19 14v4a1 1 0 0 1-1 1H6a1 1 0 0 1-1-1V6a1 1 0 0 1 1-1h4"/></svg></a></div></td><td><span class="availability-badge open">Open</span></td><td class="combined-score">38.10</td><td>28.09</td><td>27.18</td><td>49.06</td><td>87.64</td><td>35.78</td></tr>
<tr><td class="rank">#25</td><td><div class="model-cell"><span class="model-name">facebook/omniASR-CTC-3B</span><a class="model-link" href="https://huggingface.co/facebook/omniASR-CTC-3B" target="_blank" rel="noreferrer" title="Open facebook/omniASR-CTC-3B" aria-label="Open facebook/omniASR-CTC-3B model page"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M14 5h5v5m0-5-8 8M19 14v4a1 1 0 0 1-1 1H6a1 1 0 0 1-1-1V6a1 1 0 0 1 1-1h4"/></svg></a></div></td><td><span class="availability-badge open">Open</span></td><td class="combined-score">39.85</td><td>30.55</td><td>28.75</td><td>51.76</td><td>86.64</td><td>36.14</td></tr>
<tr><td class="rank">#26</td><td><div class="model-cell"><span class="model-name">facebook/omniASR-CTC-1B</span><a class="model-link" href="https://huggingface.co/facebook/omniASR-CTC-1B" target="_blank" rel="noreferrer" title="Open facebook/omniASR-CTC-1B" aria-label="Open facebook/omniASR-CTC-1B model page"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M14 5h5v5m0-5-8 8M19 14v4a1 1 0 0 1-1 1H6a1 1 0 0 1-1-1V6a1 1 0 0 1 1-1h4"/></svg></a></div></td><td><span class="availability-badge open">Open</span></td><td class="combined-score">40.50</td><td>33.78</td><td>31.01</td><td>47.05</td><td>79.37</td><td>38.35</td></tr>
<tr><td class="rank">#27</td><td><div class="model-cell"><span class="model-name">facebook/omniASR-LLM-300M</span><a class="model-link" href="https://huggingface.co/facebook/omniASR-LLM-300M" target="_blank" rel="noreferrer" title="Open facebook/omniASR-LLM-300M" aria-label="Open facebook/omniASR-LLM-300M model page"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M14 5h5v5m0-5-8 8M19 14v4a1 1 0 0 1-1 1H6a1 1 0 0 1-1-1V6a1 1 0 0 1 1-1h4"/></svg></a></div></td><td><span class="availability-badge open">Open</span></td><td class="combined-score">41.10</td><td>30.75</td><td>29.40</td><td>53.61</td><td>91.83</td><td>38.06</td></tr>
<tr><td class="rank">#28</td><td><div class="model-cell"><span class="model-name">cast/whisper-large-v3-mwa-hy</span><span class="model-origin-flag" title="Armenian-origin model" aria-label="Armenian-origin model"></span><a class="model-link" href="https://huggingface.co/Center-of-Advanced-Software-Technologies/whisper-large-v3-mwa-hy" target="_blank" rel="noreferrer" title="Open cast/whisper-large-v3-mwa-hy" aria-label="Open cast/whisper-large-v3-mwa-hy model page"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M14 5h5v5m0-5-8 8M19 14v4a1 1 0 0 1-1 1H6a1 1 0 0 1-1-1V6a1 1 0 0 1 1-1h4"/></svg></a></div></td><td><span class="availability-badge open">Open</span></td><td class="combined-score">41.41</td><td>29.59</td><td>30.69</td><td>52.89</td><td>75.04</td><td>55.75</td></tr>
<tr><td class="rank">#29</td><td><div class="model-cell"><span class="model-name">facebook/mms-1b-all</span><a class="model-link" href="https://huggingface.co/facebook/mms-1b-all" target="_blank" rel="noreferrer" title="Open facebook/mms-1b-all" aria-label="Open facebook/mms-1b-all model page"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M14 5h5v5m0-5-8 8M19 14v4a1 1 0 0 1-1 1H6a1 1 0 0 1-1-1V6a1 1 0 0 1 1-1h4"/></svg></a></div></td><td><span class="availability-badge open">Open</span></td><td class="combined-score">47.50</td><td>43.09</td><td>34.52</td><td>53.04</td><td>83.38</td><td>44.79</td></tr>
<tr><td class="rank">#30</td><td><div class="model-cell"><span class="model-name">facebook/omniASR-CTC-300M</span><a class="model-link" href="https://huggingface.co/facebook/omniASR-CTC-300M" target="_blank" rel="noreferrer" title="Open facebook/omniASR-CTC-300M" aria-label="Open facebook/omniASR-CTC-300M model page"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M14 5h5v5m0-5-8 8M19 14v4a1 1 0 0 1-1 1H6a1 1 0 0 1-1-1V6a1 1 0 0 1 1-1h4"/></svg></a></div></td><td><span class="availability-badge open">Open</span></td><td class="combined-score">50.61</td><td>44.92</td><td>40.25</td><td>57.79</td><td>86.53</td><td>47.25</td></tr>
<tr><td class="rank">#31</td><td><div class="model-cell"><span class="model-name">google/gemma-4-e4b-it</span><a class="model-link" href="https://huggingface.co/google/gemma-4-e4b-it" target="_blank" rel="noreferrer" title="Open google/gemma-4-e4b-it" aria-label="Open google/gemma-4-e4b-it model page"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M14 5h5v5m0-5-8 8M19 14v4a1 1 0 0 1-1 1H6a1 1 0 0 1-1-1V6a1 1 0 0 1 1-1h4"/></svg></a></div></td><td><span class="availability-badge open">Open</span></td><td class="combined-score">50.65</td><td>48.48</td><td>37.11</td><td>57.62</td><td>79.72</td><td>44.14</td></tr>
<tr><td class="rank">#32</td><td><div class="model-cell"><span class="model-name">google/gemma-4-e2b-it</span><a class="model-link" href="https://huggingface.co/google/gemma-4-e2b-it" target="_blank" rel="noreferrer" title="Open google/gemma-4-e2b-it" aria-label="Open google/gemma-4-e2b-it model page"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M14 5h5v5m0-5-8 8M19 14v4a1 1 0 0 1-1 1H6a1 1 0 0 1-1-1V6a1 1 0 0 1 1-1h4"/></svg></a></div></td><td><span class="availability-badge open">Open</span></td><td class="combined-score">53.46</td><td>50.44</td><td>42.76</td><td>59.88</td><td>80.19</td><td>48.92</td></tr>
<tr><td class="rank">#33</td><td><div class="model-cell"><span class="model-name">modulate-velma-2-stt-batch-multilingual-vfast</span><a class="model-link" href="https://docs.modulate.ai/api-reference/stt/batch-multilingual-vfast" target="_blank" rel="noreferrer" title="Open modulate-velma-2-stt-batch-multilingual-vfast" aria-label="Open modulate-velma-2-stt-batch-multilingual-vfast model page"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M14 5h5v5m0-5-8 8M19 14v4a1 1 0 0 1-1 1H6a1 1 0 0 1-1-1V6a1 1 0 0 1 1-1h4"/></svg></a></div></td><td><span class="availability-badge closed">Closed</span></td><td class="combined-score">64.23</td><td>62.42</td><td>52.80</td><td>63.00</td><td>107.30</td><td>53.91</td></tr>
<tr><td class="rank">#34</td><td><div class="model-cell"><span class="model-name">openai/whisper-large-v3</span><a class="model-link" href="https://huggingface.co/openai/whisper-large-v3" target="_blank" rel="noreferrer" title="Open openai/whisper-large-v3" aria-label="Open openai/whisper-large-v3 model page"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M14 5h5v5m0-5-8 8M19 14v4a1 1 0 0 1-1 1H6a1 1 0 0 1-1-1V6a1 1 0 0 1 1-1h4"/></svg></a></div></td><td><span class="availability-badge open">Open</span></td><td class="combined-score">65.12</td><td>61.67</td><td>51.84</td><td>71.10</td><td>106.17</td><td>55.25</td></tr>
</tbody>
<!-- GENERATED:LEADERBOARD END -->
</table>
</div>
<!-- GENERATED:RESULTS-DATA START -->
<script id="results-data" type="application/json">{"v0.1": "model,model_url,availability,origin,combined,,,,mcv26,,,,fleurs,,,,poems,,,,movies,,,,infocom_may_26,,,\n,,,,WER,CER,WER-norm,CER-norm,WER,CER,WER-norm,CER-norm,WER,CER,WER-norm,CER-norm,WER,CER,WER-norm,CER-norm,WER,CER,WER-norm,CER-norm,WER,CER,WER-norm,CER-norm\ngoogle/gemini-2.5-pro,https://ai.google.dev/gemini-api/docs/models/gemini-2.5-pro,Closed,,14.307,4.0955,6.4197,2.4423,9.6864,2.5433,6.3459,1.9419,8.5084,2.0004,2.9433,1.1523,24.7593,5.3359,3.9983,1.2723,38.9844,16.3223,18.8966,10.1838,11.91,3.9989,4.5688,2.7964\nhispeech/model-23012026,https://www.hispeech.ai/api.html,Closed,\ud83c\udde6\ud83c\uddf2,16.8129,4.2261,7.4986,2.2817,11.0856,2.4903,7.3486,1.761,13.7435,3.3926,5.87,1.8887,22.7285,5.4921,4.6175,1.5223,41.6997,16.3958,19.4237,9.6334,14.7849,3.3687,4.7462,1.5464\ngoogle/gemini-2.5-flash,https://ai.google.dev/gemini-api/docs/models/gemini-2.5-flash,Closed,,17.5469,4.6513,8.6718,2.8069,12.319,2.98,8.5497,2.3472,10.598,2.4786,3.9687,1.4569,28.1633,6.2153,5.6564,1.5213,46.0284,18.7923,27.1194,12.545,14.0014,3.7409,4.7969,2.2266\ntalk2edit,https://talk2edit.com,Closed,\ud83c\udde6\ud83c\uddf2,17.8337,4.2155,8.0151,2.3704,12.3693,2.6625,8.5198,2.0073,10.5202,2.3319,4.1683,1.3561,25.0855,5.5935,5.7719,1.5641,46.0725,17.6354,20.9259,10.8242,15.7326,3.1442,4.0999,1.3002\nelevenlabs/scribe_v2,https://elevenlabs.io/docs/overview/capabilities/speech-to-text,Closed,,17.9525,4.3362,8.7695,2.5805,12.0917,2.4134,7.5713,1.7013,10.0811,2.3679,4.135,1.4697,26.0273,5.8995,8.0753,2.0474,49.9515,19.3585,28.5865,13.1939,14.1467,3.574,4.6702,2.0372\nhispeech/model-01052025,https://www.hispeech.ai/api.html,Closed,\ud83c\udde6\ud83c\uddf2,17.9822,4.48,8.6709,2.6206,10.2156,2.205,6.8155,1.6571,15.5719,3.7499,8.1204,2.4814,22.9179,5.5556,4.8484,1.6561,49.8545,19.2636,26.7504,12.4295,18.2915,4.0996,7.1605,2.0652\ngoogle/gemini-3.6-flash,https://ai.google.dev/gemini-api/docs/models/gemini-3.6-flash,Closed,,19.6098,5.2995,10.1194,3.5158,16.1478,4.0149,10.1325,3.0682,11.7095,2.8982,5.2104,1.9015,25.8694,6.329,8.0753,2.5018,46.8218,18.8627,27.4971,13.34,13.3127,4.1459,5.6207,2.9026\nwavam/default,https://wav.am,Closed,\ud83c\udde6\ud83c\uddf2,20.1028,4.9044,9.3242,2.7368,12.6209,2.9858,9.3149,2.3614,13.4156,3.3471,6.3855,2.1722,30.7834,7.3266,7.5716,2.3587,53.5044,18.3362,22.9377,9.5349,16.3139,3.7337,5.0124,1.6387\nnvidia/stt_hy_fastconformer_hybrid_large_pc,https://huggingface.co/nvidia/stt_hy_fastconformer_hybrid_large_pc,Open,\ud83c\udde6\ud83c\uddf2,20.2149,5.4453,10.6333,3.4445,7.6985,1.5642,5.229,1.1576,16.3721,4.9419,8.7246,3.7623,31.32,7.5453,11.6854,3.1524,60.9451,26.3582,37.5911,18.7198,24.5024,5.9968,10.4049,3.5285\ngoogle/gemini-3.1-flash-lite,https://ai.google.dev/gemini-api/docs/models/gemini-3.1-flash-lite,Closed,,20.4652,6.2546,12.4615,4.6047,16.4105,4.615,12.9892,4.0185,11.0926,2.7537,5.1217,1.8097,26.4008,7.0928,9.702,3.5157,55.0295,26.8234,36.2734,20.9757,13.3253,3.9878,5.2215,2.6358\ntbb-asr,https://asr.tbb.rip,Closed,\ud83c\udde6\ud83c\uddf2,20.5739,5.5345,10.7268,3.4857,8.1509,1.7125,5.5219,1.29,17.0223,5.0072,8.9241,3.7525,31.2727,7.51,11.5909,3.0966,61.8708,26.5403,37.2925,18.55,24.2876,5.9792,10.2085,3.5103\nwavam/streaming,https://wav.am,Closed,\ud83c\udde6\ud83c\uddf2,21.1144,5.1478,10.3147,2.9628,13.4723,3.1688,10.1979,2.5488,15.3662,3.6728,8.3865,2.4912,31.3832,7.2507,8.3692,2.2788,54.2537,19.2146,22.7357,10.2246,17.4828,4.0445,6.3051,1.9367\nopenai/gpt-transcribe,https://developers.openai.com/api/docs/models/gpt-transcribe,Closed,,22.5595,5.9631,14.1197,4.2576,17.2917,4.1186,13.8105,3.5182,15.9275,3.9751,8.7966,2.8454,30.131,7.241,13.6531,3.5603,52.1114,23.0863,34.6042,17.7397,17.6913,4.126,7.0401,2.3361\ngoogle/gemini-3.5-flash-lite,https://ai.google.dev/gemini-api/docs/models/gemini-3.5-flash-lite,Closed,,26.2101,8.3246,17.6418,6.5826,23.6786,7.7033,19.5952,6.944,14.8049,4.1232,8.7357,3.1433,33.7402,9.2069,15.521,5.1867,56.1668,25.1706,38.9265,20.1264,17.249,5.3594,8.4152,3.8216\nchillarmo/whisper-large-v3-turbo-armenian,https://huggingface.co/chillarmo/whisper-large-v3-turbo-armenian,Open,\ud83c\udde6\ud83c\uddf2,33.1251,13.6194,23.3776,11.3725,12.9432,2.5861,9.6292,2.0609,25.0361,9.393,16.0302,7.8195,55.1113,26.9981,35.6858,22.704,82.0947,39.4393,59.5362,30.5854,49.2639,29.0766,37.3994,26.762\narampacha/whisper-large-hy-2,https://huggingface.co/arampacha/whisper-large-hy-2,Open,\ud83c\udde6\ud83c\uddf2,33.283,8.1694,22.433,5.6564,23.1886,4.7625,19.6082,4.1286,17.856,3.7587,10.9916,2.6153,44.4363,11.0406,25.5326,6.4506,77.669,32.3799,55.9431,23.5714,39.8496,8.9892,17.1662,4.6599\nfacebook/omniASR-LLM-3B,https://huggingface.co/facebook/omniASR-LLM-3B,Open,,39.2308,10.9837,16.8896,6.6404,27.3266,4.8987,9.4384,2.298,26.9201,5.4025,6.5351,2.2249,56.5558,16.7054,26.3796,9.3765,89.4516,57.1848,63.3258,48.3842,36.3303,7.5905,8.7384,2.9628\nfacebook/omniASR-LLM-1B,https://huggingface.co/facebook/omniASR-LLM-1B,Open,,39.6051,10.7221,16.9431,6.32,28.0104,5.1484,10.168,2.5503,27.4203,5.4759,7.1393,2.3076,55.7526,15.2889,25.341,7.82,89.8539,54.4434,59.6566,44.753,36.9179,7.6296,9.898,3.0229\nfacebook/omniASR-CTC-3B,https://huggingface.co/facebook/omniASR-CTC-3B,Open,,41.1224,11.8272,21.4443,7.727,30.4883,5.3088,13.5823,2.688,28.9152,5.5969,9.2789,2.4137,58.2351,14.6111,30.2455,7.018,86.9501,68.524,77.1309,64.3544,37.2275,7.6903,10.8612,3.0929\ncast/whisper-large-v3-mwa-hy,https://huggingface.co/Center-of-Advanced-Software-Technologies/whisper-large-v3-mwa-hy,Open,\ud83c\udde6\ud83c\uddf2,41.6865,10.1111,22.5511,6.1765,29.5828,5.3063,12.1997,2.6544,31.9996,7.411,13.2531,4.358,52.8174,14.2677,30.2603,8.748,75.227,31.1618,50.4261,21.9763,56.3404,14.1713,38.8252,9.9003\nfacebook/omniASR-CTC-1B,https://huggingface.co/facebook/omniASR-CTC-1B,Open,,41.9099,10.4939,21.4817,6.3275,33.8327,6.0835,17.5728,3.4618,31.1659,6.0664,12.3275,2.9041,54.0054,12.1948,22.3615,4.3528,80.0577,48.8093,63.3694,42.6041,39.3062,8.1153,13.6683,3.5268\nfacebook/omniASR-LLM-300M,https://huggingface.co/facebook/omniASR-LLM-300M,Open,,42.4713,11.4939,21.025,7.0071,30.7231,5.6727,13.4008,3.0543,29.7377,6.1456,10.0327,3.0011,59.951,16.1485,32.1653,8.7,92.9765,57.3088,65.5308,46.5174,39.1293,8.1457,13.5986,3.61\nfacebook/mms-1b-all,https://huggingface.co/facebook/mms-1b-all,Open,,48.6009,10.7795,30.4742,6.5337,43.0794,7.7882,29.2,5.2074,34.606,6.3033,16.5623,3.1892,59.6641,14.0066,30.9222,6.2569,83.5855,37.9021,69.7761,29.0887,44.8537,8.4724,21.9061,3.9048\ngoogle/gemma-4-e4b-it,https://huggingface.co/google/gemma-4-e4b-it,Open,,50.6746,18.9239,41.7823,16.2226,48.4173,17.8457,43.1284,16.1956,37.3402,12.8714,30.2644,11.3516,58.3259,20.7961,42.1031,15.6847,79.7761,42.2581,66.2479,35.4199,44.3925,15.2009,32.3554,12.8153\nfacebook/omniASR-CTC-300M,https://huggingface.co/facebook/omniASR-CTC-300M,Open,,51.743,12.7912,34.645,8.7434,44.9164,8.3903,30.996,5.8332,40.4468,7.9296,23.696,4.8357,63.633,15.2827,37.8934,7.7029,86.8801,51.8881,75.4837,46.1984,47.9497,9.7258,25.9046,5.2149\ngoogle/gemma-4-e2b-it,https://huggingface.co/google/gemma-4-e2b-it,Open,,53.5029,20.7604,44.578,17.6968,50.4164,18.8588,45.5343,17.4727,42.9199,18.2358,34.4715,13.5592,59.9042,22.157,44.7634,17.1837,80.2081,40.8992,67.8116,35.16,49.2134,18.0144,35.9103,14.9587\nopenai/whisper-large-v3,https://huggingface.co/openai/whisper-large-v3,Open,,65.131,20.8108,58.645,19.0484,61.7163,17.1442,59.0287,16.5047,51.9618,14.6363,45.4687,13.1643,71.0738,28.6474,58.7732,25.5887,105.8362,52.0858,95.3176,48.8992,55.3737,16.2242,45.8019,13.6535\n", "v1.0": "model,model_url,availability,origin,combined,,,,,,mcv26,,,,,,fleurs,,,,,,poems,,,,,,movies,,,,,,infocom_may_26,,,,,\n,,,,WER-NN,CER-NN,WER-norm-NN,CER-norm-NN,WER-LLM,CER-LLM,WER-NN,CER-NN,WER-norm-NN,CER-norm-NN,WER-LLM,CER-LLM,WER-NN,CER-NN,WER-norm-NN,CER-norm-NN,WER-LLM,CER-LLM,WER-NN,CER-NN,WER-norm-NN,CER-norm-NN,WER-LLM,CER-LLM,WER-NN,CER-NN,WER-norm-NN,CER-norm-NN,WER-LLM,CER-LLM,WER-NN,CER-NN,WER-norm-NN,CER-norm-NN,WER-LLM,CER-LLM\ngoogle/gemini-2.5-pro,https://ai.google.dev/gemini-api/docs/models/gemini-2.5-pro,Closed,,13.7974,3.5302,5.9092,2.0268,4.5053,1.7318,8.9186,1.9428,5.6918,1.4074,3.7136,1.1449,7.9099,1.5417,2.3629,0.6945,1.5637,0.5736,20.0843,4.6168,3.9517,1.2301,3.2954,1.1004,39.1937,15.8009,18.7253,10.0507,17.0430,9.2021,11.2615,3.5371,3.7985,2.3468,2.9650,2.0952\nhispeech/model-23012026,https://www.hispeech.ai/api.html,Closed,\ud83c\udde6\ud83c\uddf2,16.5563,4.0539,7.2132,2.1,6.1746,1.8900,10.939,2.4254,7.1981,1.6953,5.2458,1.4212,13.1254,2.9214,5.2218,1.4057,4.2142,1.1990,22.6937,5.4628,4.5762,1.4931,4.1560,1.3853,41.3462,16.2466,19.0857,9.4533,21.0401,10.0282,14.3616,3.0179,4.1479,1.1726,3.2761,0.9374\ngoogle/gemini-3-flash-preview,https://ai.google.dev/gemini-api/docs/models/gemini-3-flash-preview,Closed,,16.6422,4.7749,7.9643,3.0348,6.4066,2.6007,10.7675,2.7593,7.7238,2.2684,5.5339,1.9832,8.9944,1.9516,3.5555,1.1279,2.3788,0.8646,24.3595,6.1238,6.0037,2.2047,5.4416,2.2109,51.385,23.6129,24.6857,15.6069,22.7884,13.0900,11.0591,3.4362,4.0971,2.2948,3.3141,2.1197\ngoogle/gemini-2.5-flash,https://ai.google.dev/gemini-api/docs/models/gemini-2.5-flash,Closed,,17.3451,4.3127,8.4279,2.645,6.9826,2.4420,12.2269,2.9418,8.4592,2.3101,6.4525,2.0507,10.0956,2.0389,3.3827,1.0202,2.6616,0.9208,24.8392,5.5194,5.5786,1.453,4.9221,1.4009,45.8716,17.5237,26.9626,12.4985,25.1076,12.8996,13.4696,3.316,4.1288,1.8084,3.1303,1.5318\ntalk2edit,https://talk2edit.com,Closed,\ud83c\udde6\ud83c\uddf2,17.6595,4.0585,7.7697,2.205,6.4183,2.0365,12.3003,2.6328,8.4372,1.9783,6.4267,1.8675,10.0676,1.9634,3.7338,0.983,2.7892,0.8448,24.967,5.4356,5.59,1.4068,4.9853,1.2569,46.1084,17.5264,20.7106,10.6571,20.3181,10.6327,15.2853,2.7646,3.392,0.9026,2.2432,0.6491\nelevenlabs/scribe_v2,https://elevenlabs.io/docs/overview/capabilities/speech-to-text,Closed,,17.7352,4.135,8.484,2.3703,7.2094,2.1951,12.0983,2.4332,7.5592,1.7181,5.6724,1.5105,9.4471,1.8591,3.3883,0.942,2.5951,0.8163,26.0675,5.9168,8.0714,2.0723,7.5300,2.0332,49.8236,19.2328,28.4484,13.0482,27.8573,13.8125,13.2355,2.727,3.5063,1.1775,2.4080,0.8432\nhispeech/model-01052025,https://www.hispeech.ai/api.html,Closed,\ud83c\udde6\ud83c\uddf2,17.6273,4.2229,8.2688,2.3606,7.2621,2.1875,10.0462,2.1233,6.6405,1.5749,5.0587,1.4026,14.4111,2.9577,6.8602,1.6843,5.8722,1.5255,22.9573,5.5159,4.7652,1.6088,4.4918,1.5484,49.4884,18.9792,26.3736,12.123,26.3287,12.3903,17.7528,3.6525,6.5616,1.6358,5.9058,1.4367\ngoogle/gemini-3.6-flash,https://ai.google.dev/gemini-api/docs/models/gemini-3.6-flash,Closed,,19.419,5.157,9.8969,3.3693,8.1853,3.1683,16.0404,3.976,10.029,3.0278,7.8706,2.8983,11.236,2.4968,4.7091,1.5052,3.4934,1.2655,25.8303,6.2769,7.9979,2.4518,7.2887,2.4812,46.639,18.7917,27.2791,13.2479,24.2725,0.1275,12.9318,3.8096,5.1007,2.559,4.0999,2.3685\ngoogle/gemini-3.5-transcribe-preview,https://ai.google.dev/gemini-api/docs/models/gemini-3.5-transcribe,Closed,,19.5941,5.7025,11.519,4.1037,9.7072,3.7901,15.6807,4.5764,11.887,3.8725,9.6460,3.6928,10.9453,2.5197,5.3054,1.6418,4.0590,1.5272,28.039,7.6141,11.3723,3.9888,10.3741,4.0061,46.4626,20.2886,28.9582,15.5113,26.8646,15.0236,13.2608,3.5275,4.9292,2.1296,3.1874,1.2897\nwavam/default,https://wav.am,Closed,\ud83c\udde6\ud83c\uddf2,19.1325,4.2133,8.1242,2.036,6.7575,1.8354,11.418,2.2543,7.8735,1.6329,5.8202,1.3428,11.9068,2.2978,4.6255,1.1036,3.6598,1.0115,30.4112,6.8468,6.8538,1.8512,6.4963,1.8634,52.9993,17.9326,22.2857,9.0794,21.6375,9.6514,15.6713,3.2102,4.2686,1.1387,3.0543,0.8046\nnvidia/stt_hy_fastconformer_hybrid_large_pc,https://huggingface.co/nvidia/stt_hy_fastconformer_hybrid_large_pc,Open,\ud83c\udde6\ud83c\uddf2,19.855,5.0817,10.2346,3.0643,10.1566,3.1680,7.6978,1.5793,5.2203,1.1705,4.3927,1.1239,15.0036,3.5369,7.2949,2.3265,6.8648,2.6174,31.2757,7.5999,11.6767,3.2166,11.4866,3.3403,60.7798,26.2211,37.4242,18.5395,42.2911,21.0769,23.5543,5.0028,9.2168,2.5037,8.6560,2.3261\ngoogle/gemini-3.1-flash-lite,https://ai.google.dev/gemini-api/docs/models/gemini-3.1-flash-lite,Closed,,20.2732,6.1219,12.2395,4.4757,,,16.333,4.5884,12.8974,3.9904,,,10.5093,2.3266,4.5363,1.3928,,,26.0833,7.0097,9.651,3.4805,,,54.8606,26.8005,36.0967,20.9395,,,12.9191,3.7046,4.6814,2.3501,,\ntbb-asr,https://asr.tbb.rip,Closed,\ud83c\udde6\ud83c\uddf2,20.189,5.1548,10.3069,3.0875,10.1421,3.2110,8.1485,1.7307,5.5114,1.3064,4.6331,1.2508,15.5179,3.5036,7.3729,2.2156,7.1143,2.6278,31.2388,7.5557,11.5823,3.1485,11.4446,3.3459,61.6708,26.4163,37.1253,18.3825,41.2369,20.8492,23.3266,4.9732,9.0072,2.4674,8.2568,2.2771\nwavam/streaming,https://wav.am,Closed,\ud83c\udde6\ud83c\uddf2,20.1754,4.4886,9.1492,2.2941,7.4813,2.0291,12.294,2.4504,8.7735,1.834,6.3833,1.5142,14.003,2.7964,6.7878,1.597,5.1015,1.2603,31.0069,6.7663,7.6568,1.7682,7.1732,1.8433,53.7491,18.7794,22.1011,9.7298,21.2510,9.9826,16.8354,3.5291,5.5644,1.4409,4.3597,1.1315\nopenai/gpt-transcribe,https://developers.openai.com/api/docs/models/gpt-transcribe,Closed,,22.4032,5.8504,13.9229,4.142,,,17.2799,4.1466,13.7861,3.5453,,,15.367,3.5199,8.1754,2.3834,,,30.174,7.2858,13.6447,3.6129,,,51.9495,23.0243,34.4615,17.6689,,,17.2403,3.7391,6.4346,1.9513,,\nNCCAIT/model-2024,https://media.ican24.net/asr/index.php,Closed,\ud83c\udde6\ud83c\uddf2,25.1749,5.8724,11.6619,3.1299,,,16.1578,3.3002,11.2714,2.4415,,,19.4924,4.5586,8.7829,2.6809,,,33.6321,7.3734,7.62,1.799,,,61.759,23.9249,33.5385,14.8149,,,25.8256,5.006,5.3548,1.4187,,\ngoogle/gemini-3.5-flash-lite,https://ai.google.dev/gemini-api/docs/models/gemini-3.5-flash-lite,Closed,,26.0597,8.2139,17.4495,6.4669,,,23.686,7.7209,19.5678,6.9595,,,14.3161,3.7307,8.2423,2.7553,,,33.7164,9.1831,15.45,5.1471,,,56.0074,25.1268,38.7868,20.0707,,,16.7405,4.9996,7.7558,3.4649,,\ndeepgram/nova-3,https://developers.deepgram.com/docs/models-languages-overview#nova-3,Closed,,28.4625,12.7777,20.4675,10.7331,,,7.3101,3.681,5.7105,3.3149,,,33.5625,19.7212,26.6997,18.2793,,,48.35,20.5149,31.5875,16.3403,,,75.8116,42.8104,58.9626,36.357,,,36.6633,12.5891,22.1876,9.6754,,\nfacebook/seamless-m4t-v2-large,https://huggingface.co/facebook/seamless-m4t-v2-large,Open,,28.9396,11.101,20.0367,9.2159,,,15.1516,4.1544,11.7394,3.6036,,,14.4415,3.4623,7.2515,2.3667,,,43.9325,19.4122,27.8982,16.04,,,80.157,42.5012,55.0163,34.8458,,,37.4272,18.3856,27.9906,16.7714,,\nchillarmo/whisper-large-v3-turbo-armenian,https://huggingface.co/chillarmo/whisper-large-v3-turbo-armenian,Open,\ud83c\udde6\ud83c\uddf2,32.997,13.5605,23.232,11.3131,,,12.9445,2.6164,9.6248,2.0897,,,24.6912,9.1231,15.7211,7.557,,,55.087,27.0936,35.6862,22.8088,,,82.0131,39.6582,59.4901,30.8246,,,48.7979,28.5952,36.7147,26.2591,,\narampacha/whisper-large-hy-2,https://huggingface.co/arampacha/whisper-large-hy-2,Open,\ud83c\udde6\ud83c\uddf2,33.1747,8.0929,22.2623,5.5729,,,23.1566,4.7752,19.5509,4.1384,,,17.3682,3.4163,10.4603,2.2847,,,44.4808,11.076,25.5156,6.4975,,,77.8229,32.5731,55.8593,23.727,,,39.4787,8.6337,16.7058,4.2898,,\nmodulate-velma-2-stt-batch-multilingual,https://docs.modulate.ai/api-reference/stt/batch,Closed,,33.7093,8.5257,15.0712,4.8916,,,27.3541,4.9633,11.7635,2.552,,,23.9477,4.455,8.1977,1.8809,,,37.5224,8.4831,11.7082,2.9228,,,72.7329,43.5558,51.7626,36.0719,,,33.7657,6.4564,11.6941,2.5574,,\nfacebook/omniASR-LLM-3B,https://huggingface.co/facebook/omniASR-LLM-3B,Open,,37.7602,10.4169,17.0378,6.5893,,,27.4491,4.9582,9.4695,2.3086,,,26.6812,5.1467,6.2361,1.9196,,,49.8366,14.837,26.3973,9.3886,,,87.4294,55.934,67.4725,51.2743,,,35.1828,6.3915,7.2731,1.7192,,\nfacebook/omniASR-LLM-1B,https://huggingface.co/facebook/omniASR-LLM-1B,Open,,38.1027,10.1702,17.3619,6.3024,,,28.0866,5.1985,10.2105,2.5542,,,27.1787,5.2503,6.7766,2.0304,,,49.0617,13.4486,25.3372,7.8833,,,87.6411,53.2167,66.5934,47.8671,,,35.7839,6.4813,8.48,1.8282,,\nfacebook/omniASR-CTC-3B,https://huggingface.co/facebook/omniASR-CTC-3B,Open,,39.8502,11.3702,21.1284,7.4686,,,30.5487,5.372,13.5728,2.7199,,,28.7495,5.5262,9.0392,2.2999,,,51.7554,12.8105,30.2545,7.1382,,,86.6443,68.1739,76.6505,63.6294,,,36.1382,6.559,9.3947,1.9075,,\nfacebook/omniASR-CTC-1B,https://huggingface.co/facebook/omniASR-CTC-1B,Open,,40.5039,10.0225,21.1242,6.0071,,,33.7769,6.1455,17.5695,3.5091,,,31.0135,5.9871,12.082,2.7849,,,47.048,10.3318,22.2986,4.4458,,,79.3666,47.9913,62.2945,40.5049,,,38.3525,7.1015,12.3293,2.4608,,\nfacebook/omniASR-LLM-300M,https://huggingface.co/facebook/omniASR-LLM-300M,Open,,41.1005,10.9634,21.4355,7.1199,,,30.7537,5.7326,13.44,3.0853,,,29.4035,5.848,9.5965,2.6483,,,53.611,14.3716,32.1753,8.8141,,,91.8313,56.3105,72.378,51.3255,,,38.0615,7.0238,12.2658,2.4319,,\ncast/whisper-large-v3-mwa-hy,https://huggingface.co/Center-of-Advanced-Software-Technologies/whisper-large-v3-mwa-hy,Open,\ud83c\udde6\ud83c\uddf2,41.4074,9.888,22.2402,5.9401,,,29.587,5.3504,12.192,2.6882,,,30.6893,6.1425,11.8703,3.0468,,,52.894,14.3848,30.265,8.8831,,,75.0441,31.3174,50.3033,22.1481,,,55.751,13.6638,38.182,9.3864,,\nfacebook/mms-1b-all,https://huggingface.co/facebook/mms-1b-all,Open,,47.5016,10.5131,30.3636,6.5055,,,43.0907,7.8854,29.1964,5.2982,,,34.5184,6.1032,16.323,2.98,,,53.0364,12.1149,30.8318,6.3091,,,83.3804,37.8769,69.3626,28.9845,,,44.7931,8.237,21.5842,3.63,,\ngoogle/gemma-4-e4b-it,https://huggingface.co/google/gemma-4-e4b-it,Open,,50.6515,18.8821,41.6963,16.2657,,,48.4772,17.9372,43.1154,16.3293,,,37.1122,12.6134,29.971,11.1215,,,57.6226,20.5715,42.1149,15.8227,,,79.7195,42.3247,66.2418,35.6315,,,44.1415,15.0357,32.0015,12.6762,,\nfacebook/omniASR-CTC-300M,https://huggingface.co/facebook/omniASR-CTC-300M,Open,,50.609,12.4134,34.4176,8.5091,,,44.9247,8.5184,31.0132,5.9384,,,40.2482,7.8284,23.4173,4.6901,,,57.7912,13.5043,37.9008,7.8377,,,86.5296,51.3048,74.9978,44.3746,,,47.2479,8.9582,24.9762,4.4029,,\ngoogle/gemma-4-e2b-it,https://huggingface.co/google/gemma-4-e2b-it,Open,,53.4564,20.761,44.744,17.8728,,,50.438,18.9471,46.0006,17.8067,,,42.7637,18.0419,34.3792,13.4502,,,59.884,22.2016,44.7652,17.3689,,,80.187,41.0153,67.8505,35.4745,,,48.9181,17.8227,35.5714,14.8339,,\nmodulate-velma-2-stt-batch-multilingual-vfast,https://docs.modulate.ai/api-reference/stt/batch-multilingual-vfast,Closed,,64.2304,20.1585,57.5832,18.3732,,,62.4152,17.6283,59.6782,17.012,,,52.8034,14.8276,46.4501,13.3834,,,62.9995,22.3042,50.7059,19.0541,,,107.3042,53.6916,95.6396,50.332,,,53.9099,14.8497,43.9878,12.2361,,\nopenai/whisper-large-v3,https://huggingface.co/openai/whisper-large-v3,Open,,65.1203,20.8967,58.621,19.1417,,,61.6696,17.3075,58.9672,16.6797,,,51.8363,14.5561,45.5082,13.098,,,71.1017,28.7484,58.772,25.7018,,,106.1662,52.4882,95.3758,49.3027,,,55.2512,16.0189,45.652,13.4532,,\n"}</script>
<!-- GENERATED:RESULTS-DATA END -->
<div class="empty-state" id="empty-state" hidden>
<h3>No matching models</h3>
<p>Select Open or Closed in the Availability column to show models.</p>
</div>
</div>
<section class="comparison-chart" aria-labelledby="comparison-chart-title">
<div class="comparison-chart-heading">
<div>
<h2 id="comparison-chart-title">Model comparison</h2>
<p>Compare model error rates for one dataset and metric. Lower is better.</p>
</div>
<div class="comparison-chart-controls" aria-label="Model comparison filters"><label><span>Dataset</span><select id="chart-dataset" aria-label="Chart dataset"></select></label><label><span>Metric</span><select id="chart-metric" aria-label="Chart metric"></select></label><label><span>Availability</span><select id="chart-availability" aria-label="Chart model availability">
<option value="All">All models</option>
<option value="Open">Open only</option>
<option value="Closed">Closed only</option>
</select></label></div>
</div>
<div class="comparison-chart-card">
<div class="comparison-bars" id="comparison-bars" aria-live="polite"></div>
<p class="comparison-chart-empty" id="comparison-chart-empty" hidden>No models match the current filters.</p>
</div>
</section>
<section class="citation-card" aria-labelledby="citation-title">
<p class="citation-label" id="citation-title">Cite ArmBench-ASR</p>
<p>If you use ArmBench-ASR in your research, please cite it as follows:</p>
<pre><code>@misc{armbench-asr,
title={ArmBench-ASR: Benchmarking Speech-to-Text Models on Armenian},
author={Metric-AI-Lab},
year={2026},
howpublished={\url{https://metric-ai-armbench-asr.static.hf.space}},
note={Benchmark for Evaluating Speech-to-Text Models on Armenian}
}</code></pre>
</section>
</section>
<section class="panel" id="datasets" role="tabpanel" hidden>
<div class="section-intro">
<h2>Five datasets, one Armenian benchmark</h2>
<p>10,113 clips and 20.7 hours of speech spanning read, literary, cinematic, and conversational Armenian. Every model is evaluated on the same fixed test material.</p>
</div>
<div class="dataset-grid">
<article class="dataset-card">
<div class="dataset-card-top"><span>01</span><span class="dataset-tag public">Public</span></div>
<h3><a class="dataset-source" href="https://mozilladatacollective.com/datasets/cmqinsp2y00xsnr07ra8gxfoi" target="_blank" rel="noreferrer">Common Voice 26 <span>↗</span></a></h3>
<p>Mozilla Common Voice Scripted Speech 26.0 for Armenian, using the full <code>hy-AM</code> test split released in June 2026.</p>
<dl class="dataset-meta">
<div>
<dt>Samples</dt>
<dd>6,892</dd>
</div>
<div>
<dt>Audio</dt>
<dd>10.8 h</dd>
</div>
<div>
<dt>Style</dt>
<dd>Read speech</dd>
</div>
</dl>
</article>
<article class="dataset-card">
<div class="dataset-card-top"><span>02</span><span class="dataset-tag public">Public</span></div>
<h3><a class="dataset-source" href="https://huggingface.co/datasets/google/fleurs" target="_blank" rel="noreferrer">FLEURS <span>↗</span></a></h3>
<p>Google FLEURS Armenian (<code>hy_am</code>) test split. Raw transcriptions retain punctuation and capitalization for strict scoring, with some <a href="https://huggingface.co/datasets/Metric-AI/fleurs-corrections" target="_blank" rel="noreferrer">manual corrections</a> applied.</p>
<dl class="dataset-meta">
<div>
<dt>Samples</dt>
<dd>932</dd>
</div>
<div>
<dt>Audio</dt>
<dd>3.0 h</dd>
</div>
<div>
<dt>Style</dt>
<dd>Read speech</dd>
</div>
</dl>
</article>
<article class="dataset-card">
<div class="dataset-card-top"><span>03</span><span class="dataset-tag private">Private</span></div>
<h3>Poems</h3>
<p>Read Armenian poetry with background music, literary vocabulary, expressive phrasing, and text structures underrepresented in general ASR sets.</p>
<dl class="dataset-meta">
<div>
<dt>Samples</dt>
<dd>969</dd>
</div>
<div>
<dt>Audio</dt>
<dd>3.3 h</dd>
</div>
<div>
<dt>Style</dt>
<dd>Literary</dd>
</div>
</dl>
</article>
<article class="dataset-card">
<div class="dataset-card-top"><span>04</span><span class="dataset-tag private">Private</span></div>
<h3>Movies</h3>
<p>Armenian movie dialogue with conversational delivery, produced-media acoustics, and occasional background noise.</p>
<dl class="dataset-meta">
<div>
<dt>Samples</dt>
<dd>820</dd>
</div>
<div>
<dt>Audio</dt>
<dd>1.7 h</dd>
</div>
<div>
<dt>Style</dt>
<dd>Dialogue</dd>
</div>
</dl>
</article>
<article class="dataset-card">
<div class="dataset-card-top"><span>05</span><span class="dataset-tag private">Private</span></div>
<h3>News</h3>
<p>A compact Armenian news dataset recited by a speaker, designed to evaluate clear narrated speech beyond standard public test sets.</p>
<dl class="dataset-meta">
<div>
<dt>Samples</dt>
<dd>500</dd>
</div>
<div>
<dt>Audio</dt>
<dd>1.9 h</dd>
</div>
<div>
<dt>Style</dt>
<dd>Narrated news</dd>
</div>
</dl>
</article>
</div>
<section class="citation-card" aria-labelledby="citation-title">
<p class="citation-label" id="citation-title">Cite ArmBench-ASR</p>
<p>If you use ArmBench-ASR in your research, please cite it as follows:</p>
<pre><code>@misc{armbench-asr,
title={ArmBench-ASR: Benchmarking Speech-to-Text Models on Armenian},
author={Metric-AI-Lab},
year={2026},
howpublished={\url{https://metric-ai-armbench-asr.static.hf.space}},
note={Benchmark for Evaluating Speech-to-Text Models on Armenian}
}</code></pre>
</section>
</section>
<section class="panel" id="about" role="tabpanel" hidden>
<div class="about-grid">
<div class="section-intro">
<h2>Reading the benchmark</h2>
<p> Armenian ASR Benchmark compares automatic speech recognition systems on a varied Armenian evaluation suite. Every score is an error rate, so lower values are better. </p>
<p> The benchmark aims to bring clarity to the Armenian ASR landscape, make model comparisons easier, and help model creators identify strengths, gaps, and opportunities for meaningful improvement. </p>
</div>
<div class="definition-list">
<article><strong>WER</strong><span>Word error rate with number normalization.</span></article>
<article><strong>CER</strong><span>Character error rate with number normalization.</span></article>
<article><strong>WER<sub>norm</sub></strong><span>Normalized word error rate with number normalization.</span></article>
<article><strong>CER<sub>norm</sub></strong><span>Normalized character error rate with number normalization.</span></article>
<article><strong>WER<sub>LLM</sub></strong><span>Word error rate with LLM-assisted enhancements to scoring.</span></article>
<article><strong>CER<sub>LLM</sub></strong><span>Character error rate with LLM-assisted enhancements to scoring.</span></article>
</div>
</div>
<div class="note"><strong>Availability</strong>
<p><b>Open</b> means model weights are publicly available. <b>Closed</b> means access is provided through a hosted product or API.</p>
</div>
<div class="note"><strong>API settings</strong>
<p>Hosted systems use fixed model identifiers and transcription-only prompts. Evaluated Gemini runs use temperature 0, a 1,000-token output cap, and audio limited to 40 seconds. Thinking is disabled or set to the lowest applicable level: budget 0 for Gemini 2.5 Flash, budget 128 for Gemini 2.5 Pro, and <b>MINIMAL</b> for Gemini 3.1 Flash Lite, 3.5 Flash Lite, and 3.6 Flash.</p>
</div>
<div class="note"><strong>Text cleaning</strong>
<p>References and predictions are processed with the same deterministic pipeline. Number normalization is applied before scoring; normalized metrics additionally reduce orthographic and formatting differences while preserving Armenian, English, Russian letters, digits, and recognized clock times. Both <b>WER / CER</b> and <b>WER<sub>norm</sub> / CER<sub>norm</sub></b> are reported.</p>
</div>
<div class="note"><strong>LLM-assisted evaluation</strong>
<p>The evaluator compares ASR predictions against the human reference transcript without listening to the audio, assuming the reference is correct. It distinguishes genuine recognition errors from harmless formatting or equivalent-form variations; paraphrases still count as errors. <b>WER<sub>LLM</sub> / CER<sub>LLM</sub></b> are calculated from approved edits and should be treated as model-assisted estimates. These are comparable with the normalized metrics.</p>
</div>
<section class="lab-card">
<div>
<h2>About Us</h2>
<p>Armenian ASR Benchmark is a Metric AI Lab project. The applied AI research lab builds production systems across custom language and vision-language models, multilingual voice agents, Physical AI, retrieval, and analytics—while publishing frontier research and open benchmarks.</p>
</div><a href="https://metricailab.com" target="_blank" rel="noreferrer">Visit Metric AI Lab <span>↗</span></a>
</section>
<section class="acknowledgments">
<p class="acknowledgments-label">Acknowledgments</p>
<p> We gratefully acknowledge <strong>HiSpeech</strong>, <strong>Talk2Edit</strong>, <strong>TBB ASR</strong>, <strong>NCCAIT</strong>, and <strong>Wav.am</strong> for making their systems available for evaluation at no cost, and <strong>Infocom</strong> for providing Armenian news data used in this benchmark. </p>
</section>
<footer><span>Armenian ASR Benchmark · v1.0</span><span>A Metric AI Lab project · Built for transparent Armenian speech recognition evaluation.</span></footer>
</section>
<section class="panel" id="changelog" role="tabpanel" hidden>
<div class="section-intro">
<h2>Changelog</h2>
<p>Updates to benchmark methodology and reported results.</p>
</div>
<article class="note">
<strong>v1.0</strong>
<div>
<p><b>New models.</b> Added <i>Gemini 3 Flash Preview</i>, <i>Gemini 3.5 Transcribe</i>, <i>Deepgram Nova 3</i>, <i>Modulate Velma 2</i> (standard and vfast), <i>NCCAIT/model-2024</i>, and <i>Facebook SeamlessM4T v2</i> to the benchmark.</p>
<p><b>Number normalization.</b> All numbers are converted to digits before metric computation, making numeric expressions consistent across references and predictions.</p>
<p><b>Preprocessing refinements.</b> Minor normalization corrections now convert symbols written out in text to their corresponding characters before scoring, improving consistency in the evaluation pipeline.</p>
<p><b>LLM-assisted error metrics.</b> Added WER<sub>LLM</sub> and CER<sub>LLM</sub>. An LLM reviews candidate transcript mismatches to distinguish genuine recognition errors from harmless representation differences; the approved edits are then aggregated into corpus-level word and character error rates.
</div>
</article>
<article class="note">
<strong>v0.1</strong>
<div>
<p><b>Initial release.</b> The benchmark was introduced with a punctuation-normalization and Unicode NFKC preprocessing pipeline.</p>
<p><b>Normalized metrics.</b> The additional normalization step removed punctuation and converted text to lowercase. This initial version did not normalize numbers or symbols.</p>
</div>
</article>
</section>
</main>
</body>
</html>