Download index.html from olewave/fa-bench: direct link, hf CLI and curl.
- Browser
- Download file 81.2 kB
-
https://huggingface.co/spaces/olewave/fa-bench/resolve/main/index.html
- Command line
-
hf download hf://spaces/olewave/fa-bench/index.html
-
curl -L -o index.html https://huggingface.co/spaces/olewave/fa-bench/resolve/main/index.html
81.2 kB
| <html lang="en"> | |
| <head> | |
| <script>(function(){var t;try{t=localStorage.getItem('olw-theme');}catch(e){}if(t!=='light'&&t!=='dark'){t=window.matchMedia&&matchMedia('(prefers-color-scheme: dark)').matches?'dark':'light';}document.documentElement.setAttribute('data-theme',t);})();</script> | |
| <meta charset="UTF-8"> | |
| <meta name="viewport" content="width=device-width, initial-scale=1"> | |
| <title>FA-Bench Leaderboard, Timestamp Accuracy for ASR and Forced Alignment</title> | |
| <meta name="description" content="FA-Bench ranks 16 ASR systems and 15 forced aligners by how closely their word and phone timestamps match human annotation on TIMIT and Buckeye, clean and degraded."/> | |
| <link rel="canonical" href="https://olewave.com/en/leaderboard/fa-bench"> | |
| <base target="_blank"> | |
| <style> | |
| *, ::before, ::after { box-sizing: border-box; } | |
| body { margin: 0; background: #fff; color: #0A0A0A; line-height: 1.5; -webkit-text-size-adjust: 100%; } | |
| .container { width: 100%; max-width: 1240px; margin: 0 auto; padding: 0 15px; } | |
| img, svg { vertical-align: middle; } | |
| table { border-collapse: collapse; } | |
| a { text-decoration: none; } | |
| .blog-article { background-image: none ; } | |
| .fab-mirror { display: flex; align-items: center; justify-content: space-between; gap: 12px; flex-wrap: wrap; | |
| max-width: 1240px; margin: 0 auto; padding: 18px 15px 0; font-size: 14px; } | |
| .fab-mirror span a { color: var(--olw-link, #5f3bff); } | |
| </style> | |
| <link rel="stylesheet" href="overrides.css"> | |
| <link rel="stylesheet" href="dark.css"> | |
| <link rel="stylesheet" href="leaderboard.css"> | |
| <style> | |
| .fab-page .fab-cmp { overflow-x: auto; } | |
| .fab-page .blog-article__meta a { color: #0A0A0A; } | |
| /* The header spans the table below it. It takes the card's width and place, | |
| and insets its text by the card's padding and border, so its left edge | |
| lines up with the table's. */ | |
| .fab-page .blog-article__head { box-sizing: border-box; width: min(1240px, calc(100vw - 48px)); | |
| max-width: none; margin-inline-start: calc(50% - min(620px, 50vw - 24px)); padding-left: 21px; padding-right: 21px; } | |
| .fab-page .blog-article__head .blog-article__deck { max-width: none; } | |
| /* The way into the methodology, a step larger than the deck above it. */ | |
| .fab-page .fab-page__overview { margin: 16px 0 0; font-size: 18px; line-height: 1.4; font-weight: 600; } | |
| .fab-page .fab-page__overview a { color: #5f3bff; } | |
| @media (max-width: 480px) { | |
| .fab-page .fab-cmp th, .fab-page .fab-cmp td { padding: 6px 6px; font-size: 14px; } | |
| .fab-page .fab-page__overview { font-size: 17px; } | |
| /* The card's padding narrows to 14px here, so the header's inset does too. */ | |
| .fab-page .blog-article__head { padding-left: 15px; padding-right: 15px; } | |
| } | |
| </style> | |
| </head> | |
| <body> | |
| <header class="site-header fab-mirror"> | |
| <div class="logo-header"><a class="olw-brand" href="https://olewave.com/en/" aria-label="Olewave"> | |
| <img class="olw-brand__mark" src="logos/olewave-mark.png" alt="" width="36" height="36"> | |
| <img class="olw-brand__mark olw-brand__mark--night" src="logos/olewave-mark-night.png" alt="" width="36" height="36"> | |
| <img class="olw-brand__wordmark" src="logos/olewave-wordmark.png" alt="Olewave" width="132" height="22"> | |
| </a></div> | |
| <span>A copy of <a href="https://olewave.com/en/leaderboard/fa-bench">olewave.com/en/leaderboard/fa-bench</a>, taken October 30, 2026.</span> | |
| </header> | |
| <div class="page-content bg-white"> | |
| <article class="blog-article fab-page"> | |
| <div class="container"> | |
| <div class="blog-article__inner"> | |
| <header class="blog-article__head"> | |
| <span class="blog-article__eyebrow">Speech timestamp benchmark</span> | |
| <h1 class="blog-article__title">FA-Bench Leaderboard</h1> | |
| <p class="blog-article__deck"><a href="https://github.com/olewave/fa-bench/">FA-Bench</a> measures how closely word and phone timestamps land on the boundaries human linguists placed.</p> | |
| <div class="blog-article__meta"> | |
| <span>Updated October 28, 2026</span> | |
| <span class="blog-article__sep">·</span> | |
| <a href="https://github.com/olewave/fa-bench/">github.com/olewave/fa-bench</a> | |
| <span class="blog-article__sep">·</span> | |
| <a href="https://arxiv.org/abs/2609.32396">Paper on arXiv</a> | |
| </div> | |
| <p class="fab-page__overview"><a href="https://olewave.com/en/leaderboard/fa-bench/overview">How FA-Bench scores, and the test sets</a> →</p> | |
| </header> | |
| <div class="blog-article__body"> | |
| <div class="fab-lb fab-lb--wide" data-fab-leaderboard data-logos="logos/" data-fabench-href="https://github.com/olewave/fa-bench/" data-olign-href="https://olewave.com/en/olign-olewaves-lancet-accurate-speech-to-text-forced-alignment-service/" data-withdrawn="olewave" data-withdrawn-note="Olign no longer takes part in FA-Bench. Its last results, from October 2026, are listed unranked at the bottom of the table."> | |
| <script>document.currentScript.parentElement.classList.add('is-js');</script> | |
| <script type="text/csv" data-board="asr-word" data-track="2" data-tier="word" data-href="https://olewave.com/media/blog/fa-bench-leaderboard/track2_word_f1.csv?v=1793210021" data-note="Every row is one system as shipped. A cascade of a recognizer and a forced aligner is not ranked here. It shows on the forced alignment boards under Recognized words, and in the F1 Drop of its aligner's row. Olign's price is $0.18 an hour, which is its launch promotion price. Its list price is $0.21."> | |
| rank,rank_lo,rank_hi,label,system,version,org,access,pipeline,family,f1_mean4,f1_mean4_lo,f1_mean4_hi,f1_mean4_recognized,f1_drop,price_usd_per_hour,f1_timit_clean,f1_timit_clean_lo,f1_timit_clean_hi,f1_timit_noisy,f1_timit_noisy_lo,f1_timit_noisy_hi,f1_buckeye_clean,f1_buckeye_clean_lo,f1_buckeye_clean_hi,f1_buckeye_noisy,f1_buckeye_noisy_lo,f1_buckeye_noisy_hi,mae_ms_timit_clean,mae_ms_timit_noisy,mae_ms_buckeye_clean,mae_ms_buckeye_noisy,wer_pct_timit_clean,wer_pct_timit_noisy,wer_pct_buckeye_clean,wer_pct_buckeye_noisy,flag,note | |
| 1,1,1,Google Chirp 2,Google Chirp 2,"chirp_2, Speech-to-Text API v2",google,commercial,one-step,API,0.486,0.475,0.498,,,0.96,0.488,0.462,0.514,0.497,0.476,0.519,0.492,0.487,0.497,0.469,0.465,0.473,32.7,32.6,26.9,28.1,4.2,6.7,13.1,18.5,, | |
| 2,2,3,CrisperWhisper,CrisperWhisper,,nyra,open,one-step,Attention,0.455,0.442,0.466,,,,0.484,0.459,0.508,0.472,0.448,0.494,0.463,0.455,0.471,0.399,0.393,0.404,33.5,36.2,37.6,41.0,3.6,4.3,11.5,15.6,⚠,"CrisperWhisper's authors used TIMIT to build its word timing and don't say which TIMIT files. FA-Bench tests on TIMIT's test files, so if the authors used those too, its TIMIT scores come from audio it has already seen." | |
| 3,2,3,Google Chirp 3,Google Chirp 3,"chirp_3, Speech-to-Text API v2",google,commercial,one-step,API,0.452,0.442,0.462,,,0.96,0.460,0.438,0.484,0.438,0.418,0.458,0.479,0.474,0.483,0.431,0.422,0.439,47.2,47.2,224.6,61.3,5.2,6.3,13.5,20.7,‡,Chirp 3 ends the last word of 15% of clean Buckeye test utterances more than 0.3 s after the audio does. | |
| 4,4,4,Qwen3-ASR-Flash,Qwen3-ASR-Flash,"qwen3-asr-flash-filetrans, DashScope API v1",alibaba,commercial,one-step,API,0.363,0.353,0.374,,,0.13,0.398,0.377,0.419,0.355,0.335,0.376,0.376,0.372,0.381,0.322,0.318,0.327,34.3,48.8,39.5,59.6,1.8,2.1,10.1,13.7,, | |
| 5,5,5,Mistral Voxtral Mini,Mistral Voxtral Mini,"voxtral-mini-2602, API v1",mistral,commercial,one-step,API,0.308,0.298,0.319,,,0.18,0.315,0.295,0.336,0.294,0.275,0.314,0.325,0.321,0.329,0.299,0.295,0.303,37.7,39.1,38.1,39.4,2.4,3.6,13.4,18.2,, | |
| 6,6,7,ElevenLabs Scribe v2,ElevenLabs Scribe v2,"scribe_v2, API v1",elevenlabs,commercial,one-step,API,0.281,0.275,0.288,,,0.22,0.304,0.289,0.319,0.297,0.284,0.310,0.267,0.265,0.270,0.256,0.254,0.259,31.5,32.5,34.6,35.9,2.1,3.2,10.7,14.1,, | |
| 7,6,7,Amazon Transcribe,Amazon Transcribe,"en-US, batch API 2017-10-26",amazon,commercial,one-step,API,0.273,0.263,0.284,,,0.36,0.259,0.239,0.281,0.264,0.244,0.285,0.289,0.285,0.293,0.281,0.277,0.284,55.7,56.6,52.8,53.7,2.7,2.9,10.3,12.3,, | |
| 8,8,9,AssemblyAI Universal-3.5 Pro,AssemblyAI Universal 3.5,"universal-3-5-pro, API v2",assemblyai,commercial,one-step,API,0.213,0.204,0.222,,,0.21,0.237,0.217,0.256,0.218,0.201,0.237,0.211,0.208,0.215,0.185,0.182,0.188,47.4,52.6,55.4,66.0,2.0,2.4,9.0,12.2,, | |
| 9,8,9,TorchAudio (ASR),TorchAudio (ASR),,pytorch,open,one-step,CTC,0.209,0.203,0.216,,,,0.254,0.240,0.270,0.213,0.201,0.226,0.211,0.208,0.214,0.159,0.157,0.161,37.8,38.2,37.8,39.2,10.8,25.6,27.8,47.0,, | |
| 10,10,11,Deepgram Nova-3,Deepgram Nova-3,"nova-3, build 2025-07-31.0, API v1",deepgram,commercial,one-step,API,0.170,0.161,0.178,,,0.26,0.191,0.173,0.208,0.185,0.168,0.202,0.155,0.152,0.158,0.148,0.145,0.151,67.6,70.2,97.8,93.3,2.4,4.4,10.8,16.8,‡,Deepgram ends the last word of 24% of clean Buckeye test utterances more than 0.3 s after the audio does. | |
| 11,10,11,Parakeet-TDT,Parakeet-TDT,,nvidia,open,one-step,Transducer,0.164,0.155,0.172,,,,0.172,0.153,0.190,0.174,0.158,0.190,0.156,0.152,0.159,0.153,0.150,0.156,79.3,77.1,80.7,80.5,2.4,3.5,11.0,15.2,, | |
| 12,12,12,IBM Watson STT en-US,IBM Watson Large,"en-US large speech model, API v1",ibm,commercial,one-step,API,0.149,0.142,0.157,,,1.27,0.135,0.121,0.151,0.132,0.118,0.147,0.176,0.172,0.179,0.155,0.152,0.158,72.4,75.9,62.0,65.0,5.1,9.2,12.4,19.9,, | |
| 13,13,14,Azure AI Speech,Azure AI Speech,"en-US, short audio REST API v1",microsoft,commercial,one-step,API,0.100,0.094,0.107,,,1.00,0.117,0.103,0.132,0.112,0.101,0.124,0.089,0.086,0.091,0.083,0.081,0.085,64.4,67.4,84.7,92.0,3.4,4.6,11.5,15.8,, | |
| 14,13,14,Speechmatics Enhanced,Speechmatics enhanced,"enhanced, builds 16.0.0 and 16.1.0, API v2",speechmatics,commercial,one-step,API,0.097,0.091,0.103,,,0.75,0.110,0.095,0.125,0.088,0.077,0.099,0.107,0.104,0.110,0.083,0.081,0.086,77.1,81.5,74.2,81.6,2.5,3.4,11.4,14.3,, | |
| 15,15,16,Whisper large-v3,Whisper large-v3,,openai,open,one-step,Attention,0.049,0.046,0.052,,,,0.029,0.023,0.036,0.029,0.023,0.036,0.067,0.065,0.069,0.069,0.068,0.071,148.0,151.8,122.7,123.0,2.9,3.1,13.7,17.5,, | |
| 16,15,16,Whisper-timestamped,Whisper-timestamped,,linto,open,one-step,Attention,0.049,0.046,0.052,,,,0.021,0.014,0.027,0.022,0.017,0.028,0.075,0.073,0.077,0.076,0.074,0.078,161.6,163.9,133.3,136.0,2.5,3.4,13.4,17.9,, | |
| ,,,Olewave Olign v1,Olign 1.0,"olign-1.0, Olign API v1",olewave,commercial,audio only,API,0.681,0.673,0.690,,,0.18,0.786,0.767,0.803,0.695,0.677,0.713,0.660,0.656,0.665,0.585,0.580,0.590,18.4,28.2,26.9,33.6,2.2,3.2,10.2,14.3,, | |
| </script> | |
| <script type="text/csv" data-board="fa-word" data-track="1" data-tier="word" data-href="https://olewave.com/media/blog/fa-bench-leaderboard/track1_word_f1.csv?v=1793210021" data-note="The Transcript switch shows each aligner handed the words Qwen3-ASR recognized in place of the reference transcript. Those words have a WER of 2.0% on TIMIT core test and 10.2% on Buckeye test on clean audio. NeuFA was not run on recognized words. MFA 3.4 on Parakeet-TDT's words scores 0.625, and WhisperX on Whisper large-v3's words scores 0.234. Olign's price is $0.18 an hour, which is its launch promotion price. Its list price is $0.21."> | |
| rank,rank_lo,rank_hi,label,system,version,org,access,pipeline,family,f1_mean4,f1_mean4_lo,f1_mean4_hi,f1_mean4_recognized,f1_drop,price_usd_per_hour,f1_timit_clean,f1_timit_clean_lo,f1_timit_clean_hi,f1_timit_noisy,f1_timit_noisy_lo,f1_timit_noisy_hi,f1_buckeye_clean,f1_buckeye_clean_lo,f1_buckeye_clean_hi,f1_buckeye_noisy,f1_buckeye_noisy_lo,f1_buckeye_noisy_hi,mae_ms_timit_clean,mae_ms_timit_noisy,mae_ms_buckeye_clean,mae_ms_buckeye_noisy,wer_pct_timit_clean,wer_pct_timit_noisy,wer_pct_buckeye_clean,wer_pct_buckeye_noisy,flag,note | |
| 1,1,1,MFA 3.4,MFA 3.4,,mfa,open,forced alignment,HMM,0.658,0.650,0.667,0.628,0.030,,0.685,0.665,0.704,0.649,0.631,0.667,0.706,0.702,0.711,0.594,0.589,0.599,21.8,30.2,21.3,35.6,,,,,, | |
| 2,2,2,PocketSphinx,PocketSphinx,,cmusphinx,open,forced alignment,HMM,0.648,0.639,0.658,0.615,0.033,,0.645,0.622,0.666,0.594,0.576,0.612,0.728,0.723,0.732,0.627,0.623,0.631,30.5,38.2,22.1,35.6,,,,,, | |
| 3,3,3,NeuFA,NeuFA,"trained with its open-source code on GitHub, not released",neufa,open,forced alignment,Attention,0.599,0.589,0.607,,,,0.600,0.578,0.621,0.493,0.475,0.511,0.712,0.707,0.717,0.589,0.584,0.594,56.9,88.6,33.9,59.1,,,,,⚠,"NeuFA was trained on 7 of the 8 speakers in FA-Bench's Buckeye test set, so most of its Buckeye test audio comes from speakers it was trained on." | |
| 4,4,4,Charsiu,Charsiu,,charsiu,open,forced alignment,Frame,0.563,0.553,0.573,0.539,0.024,,0.597,0.574,0.620,0.532,0.512,0.550,0.627,0.622,0.632,0.497,0.492,0.502,28.1,40.2,28.1,60.3,,,,,, | |
| 5,5,5,MAPS,MAPS,,maps,open,forced alignment,Frame,0.547,0.537,0.556,0.523,0.024,,0.712,0.690,0.734,0.412,0.393,0.432,0.642,0.638,0.647,0.421,0.416,0.425,25.5,110.0,37.9,111.4,,,,,⚠,"MAPS was trained on 7 of the 8 speakers in FA-Bench's Buckeye test set, so most of its Buckeye test audio comes from speakers it was trained on." | |
| 6,6,6,CrisperWhisper,CrisperWhisper,,nyra,open,forced alignment,Attention,0.473,0.461,0.485,0.456,0.017,,0.489,0.464,0.512,0.480,0.456,0.502,0.491,0.483,0.499,0.434,0.428,0.441,34.1,36.9,43.1,52.5,,,,,⚠,"CrisperWhisper's authors used TIMIT to build its word timing and don't say which TIMIT files. FA-Bench tests on TIMIT's test files, so if the authors used those too, its TIMIT scores come from audio it has already seen." | |
| 7,7,7,Meta UnitY2,UnitY2,,meta,open,forced alignment,Frame,0.439,0.428,0.449,0.417,0.022,,0.380,0.357,0.403,0.393,0.373,0.412,0.495,0.490,0.500,0.486,0.482,0.491,52.2,53.8,40.6,49.0,,,,,, | |
| 8,8,8,Qwen3-FA,Qwen3-FA,,alibaba,open,forced alignment,Attention,0.402,0.391,0.413,0.389,0.013,,0.416,0.394,0.438,0.386,0.365,0.407,0.426,0.422,0.430,0.380,0.376,0.384,32.2,38.5,33.8,50.8,,,,,, | |
| 9,9,9,Meta MMS-FA,MMS-FA,,meta,open,forced alignment,CTC,0.345,0.338,0.352,0.331,0.014,,0.353,0.339,0.367,0.350,0.337,0.364,0.344,0.341,0.347,0.332,0.329,0.335,31.1,31.5,37.1,41.2,,,,,, | |
| 10,10,10,ElevenLabs-FA,ElevenLabs FA,"forced-alignment, API v1",elevenlabs,commercial,forced alignment,API,0.295,0.289,0.302,0.280,0.015,0.22,0.299,0.285,0.313,0.300,0.288,0.313,0.293,0.290,0.296,0.288,0.286,0.291,31.9,32.8,35.5,38.6,,,,,, | |
| 11,11,13,BFA,BFA,,bfa,open,forced alignment,CTC,0.272,0.264,0.280,0.263,0.009,,0.276,0.258,0.294,0.255,0.240,0.269,0.285,0.282,0.289,0.272,0.269,0.275,51.0,56.7,57.2,61.7,,,,,, | |
| 12,11,13,WhisperX,WhisperX,,whisperx,open,forced alignment,CTC,0.271,0.264,0.277,0.257,0.014,,0.280,0.266,0.294,0.267,0.255,0.279,0.281,0.278,0.284,0.254,0.252,0.257,37.2,39.5,41.7,52.0,,,,,, | |
| 13,12,13,TorchAudio,TorchAudio,,pytorch,open,forced alignment,CTC,0.265,0.258,0.272,0.252,0.013,,0.273,0.258,0.289,0.255,0.243,0.269,0.275,0.272,0.278,0.256,0.253,0.258,39.8,42.4,42.4,51.4,,,,,, | |
| 14,14,14,stable-ts,stable-ts,,stablets,open,forced alignment,Attention,0.251,0.243,0.260,0.236,0.015,,0.197,0.180,0.215,0.210,0.195,0.226,0.301,0.297,0.305,0.297,0.294,0.301,96.8,94.5,71.2,75.2,,,,,, | |
| 15,15,15,NVIDIA NeMo-FA,NeMo-FA 80 ms,,nvidia,open,forced alignment,CTC,0.155,0.149,0.163,0.147,0.008,,0.157,0.143,0.173,0.158,0.145,0.172,0.151,0.148,0.154,0.156,0.153,0.159,79.2,79.7,91.4,93.1,,,,,†,NeMo-FA here is its 80 ms model. The 40 ms model's mean F1 is 0.199. | |
| ,,,Olewave Olign v1,Olign 1.0,"olign-1.0, Olign API v1",olewave,commercial,forced alignment,API,0.735,0.726,0.743,,,0.18,0.805,0.787,0.823,0.710,0.691,0.727,0.755,0.751,0.759,0.668,0.664,0.673,18.2,33.0,22.4,43.8,,,,,, | |
| </script> | |
| <script type="text/csv" data-board="fa-phone" data-track="1" data-tier="phone" data-href="https://olewave.com/media/blog/fa-bench-leaderboard/track1_phone_f1.csv?v=1793210021" data-note="FALCON is left off this board because it is handed the reference phones, which no other aligner here gets. Its mean F1 is 0.560. The Transcript switch shows each aligner handed the words Qwen3-ASR recognized in place of the reference transcript. Those words have a WER of 2.0% on TIMIT core test and 10.2% on Buckeye test on clean audio. NeuFA was not run on recognized words. MFA 3.4 on Parakeet-TDT's words scores 0.429."> | |
| rank,rank_lo,rank_hi,label,system,version,org,access,pipeline,family,f1_mean4,f1_mean4_lo,f1_mean4_hi,f1_mean4_recognized,f1_drop,price_usd_per_hour,f1_timit_clean,f1_timit_clean_lo,f1_timit_clean_hi,f1_timit_noisy,f1_timit_noisy_lo,f1_timit_noisy_hi,f1_buckeye_clean,f1_buckeye_clean_lo,f1_buckeye_clean_hi,f1_buckeye_noisy,f1_buckeye_noisy_lo,f1_buckeye_noisy_hi,mae_ms_timit_clean,mae_ms_timit_noisy,mae_ms_buckeye_clean,mae_ms_buckeye_noisy,per_pct_timit_clean,per_pct_timit_noisy,per_pct_buckeye_clean,per_pct_buckeye_noisy,flag,note | |
| 1,1,1,MFA 3.4,MFA 3.4,,mfa,open,forced alignment,HMM,0.436,0.431,0.443,0.428,0.008,,0.393,0.382,0.404,0.372,0.360,0.384,0.518,0.514,0.522,0.463,0.459,0.467,11.6,17.0,15.3,26.1,34.8,35.0,26.1,29.5,, | |
| 2,2,2,PocketSphinx,PocketSphinx,,cmusphinx,open,forced alignment,HMM,0.402,0.396,0.408,0.395,0.007,,0.338,0.326,0.350,0.326,0.314,0.337,0.494,0.490,0.497,0.450,0.447,0.454,16.4,20.3,17.1,27.3,33.9,34.3,26.0,27.1,, | |
| 3,3,3,MAPS,MAPS,,maps,open,forced alignment,Frame,0.383,0.377,0.389,0.378,0.005,,0.454,0.441,0.468,0.319,0.308,0.330,0.445,0.441,0.448,0.313,0.310,0.316,15.3,89.6,22.9,104.9,30.7,30.7,34.9,34.9,⚠,"MAPS was trained on 7 of the 8 speakers in FA-Bench's Buckeye test set, so most of its Buckeye test audio comes from speakers it was trained on." | |
| 4,4,4,Charsiu,Charsiu,,charsiu,open,forced alignment,Frame,0.372,0.367,0.378,0.369,0.003,,0.360,0.348,0.373,0.347,0.336,0.358,0.424,0.420,0.428,0.359,0.356,0.362,19.9,28.3,21.1,50.4,30.3,31.1,31.7,34.8,, | |
| 5,5,5,NeuFA,NeuFA,"trained with its open-source code on GitHub, not released",neufa,open,forced alignment,Attention,0.366,0.360,0.371,,,,0.326,0.314,0.339,0.277,0.266,0.287,0.460,0.456,0.463,0.399,0.395,0.402,20.5,56.1,22.6,46.5,36.0,39.1,28.0,30.7,⚠,"NeuFA was trained on 7 of the 8 speakers in FA-Bench's Buckeye test set, so most of its Buckeye test audio comes from speakers it was trained on." | |
| 6,6,6,TorchAudio,TorchAudio,,pytorch,open,forced alignment,CTC,0.235,0.230,0.240,0.230,0.005,,0.208,0.198,0.218,0.206,0.196,0.215,0.266,0.263,0.269,0.261,0.258,0.264,33.8,33.3,34.5,41.3,35.9,35.9,28.5,28.5,, | |
| 7,7,7,BFA,BFA,,bfa,open,forced alignment,CTC,0.217,0.212,0.221,0.213,0.004,,0.183,0.174,0.193,0.178,0.170,0.186,0.255,0.252,0.258,0.251,0.249,0.254,45.1,48.9,50.2,54.8,36.2,36.2,32.1,32.2,, | |
| ,,,Olewave Olign v1,Olign 1.0,"olign-1.0, Olign API v1",olewave,commercial,forced alignment,API,0.452,0.446,0.459,,,0.18,0.410,0.396,0.422,0.393,0.381,0.406,0.523,0.519,0.527,0.483,0.479,0.487,8.4,13.1,12.7,24.1,35.4,35.5,28.2,28.5,, | |
| </script> | |
| <script type="text/csv" data-board="fa-word-recognized" data-track="1" data-tier="word" data-href="https://olewave.com/media/blog/fa-bench-leaderboard/track1_word_f1_recognized.csv?v=1793210021" data-words="recognized" data-note="Qwen3-ASR's words have a WER of 2.0% on TIMIT core test and 10.2% on Buckeye test on clean audio. NeuFA was not run on recognized words. Olign v1 was not run on recognized words, so the unranked Olign row here is Olign 0.9."> | |
| rank,rank_lo,rank_hi,label,system,version,org,access,pipeline,family,f1_mean4,f1_mean4_lo,f1_mean4_hi,f1_mean4_recognized,f1_drop,price_usd_per_hour,f1_timit_clean,f1_timit_clean_lo,f1_timit_clean_hi,f1_timit_noisy,f1_timit_noisy_lo,f1_timit_noisy_hi,f1_buckeye_clean,f1_buckeye_clean_lo,f1_buckeye_clean_hi,f1_buckeye_noisy,f1_buckeye_noisy_lo,f1_buckeye_noisy_hi,mae_ms_timit_clean,mae_ms_timit_noisy,mae_ms_buckeye_clean,mae_ms_buckeye_noisy,wer_pct_timit_clean,wer_pct_timit_noisy,wer_pct_buckeye_clean,wer_pct_buckeye_noisy,flag,note | |
| 1,1,1,MFA 3.4,MFA 3.4,,mfa,open,forced alignment,HMM,0.628,0.619,0.637,,0.030,,0.677,0.656,0.697,0.642,0.624,0.660,0.649,0.644,0.654,0.545,0.540,0.549,21.8,29.6,22.7,33.9,,,,,, | |
| 2,2,2,PocketSphinx,PocketSphinx,,cmusphinx,open,forced alignment,HMM,0.615,0.605,0.624,,0.033,,0.632,0.610,0.653,0.585,0.567,0.602,0.669,0.665,0.674,0.573,0.568,0.577,30.6,37.5,22.6,32.7,,,,,, | |
| 3,3,3,Charsiu,Charsiu,,charsiu,open,forced alignment,Frame,0.539,0.529,0.550,,0.024,,0.587,0.564,0.609,0.522,0.503,0.541,0.586,0.581,0.591,0.463,0.459,0.468,28.2,39.9,26.1,50.9,,,,,, | |
| 4,4,4,MAPS,MAPS,,maps,open,forced alignment,Frame,0.523,0.514,0.532,,0.024,,0.702,0.679,0.724,0.407,0.388,0.427,0.595,0.591,0.600,0.387,0.382,0.392,25.5,107.8,39.8,109.2,,,,,⚠,"MAPS was trained on 7 of the 8 speakers in FA-Bench's Buckeye test set, so most of its Buckeye test audio comes from speakers it was trained on." | |
| 5,5,5,CrisperWhisper,CrisperWhisper,,nyra,open,forced alignment,Attention,0.456,0.443,0.468,,0.017,,0.485,0.459,0.509,0.472,0.449,0.495,0.464,0.456,0.471,0.403,0.397,0.408,33.8,36.8,39.2,44.4,,,,,⚠,"CrisperWhisper's authors used TIMIT to build its word timing and don't say which TIMIT files. FA-Bench tests on TIMIT's test files, so if the authors used those too, its TIMIT scores come from audio it has already seen." | |
| 6,6,6,Meta UnitY2,UnitY2,,meta,open,forced alignment,Frame,0.417,0.406,0.428,,0.022,,0.375,0.352,0.398,0.385,0.365,0.405,0.461,0.456,0.466,0.447,0.442,0.451,52.4,54.0,39.8,44.9,,,,,, | |
| 7,7,7,Qwen3-FA,Qwen3-FA,,alibaba,open,forced alignment,Attention,0.389,0.377,0.399,,0.013,,0.408,0.385,0.431,0.383,0.362,0.405,0.405,0.400,0.409,0.358,0.354,0.362,32.6,38.3,30.6,41.7,,,,,, | |
| 8,8,8,Meta MMS-FA,MMS-FA,,meta,open,forced alignment,CTC,0.331,0.324,0.337,,0.014,,0.349,0.335,0.362,0.346,0.332,0.358,0.323,0.320,0.326,0.305,0.302,0.308,31.1,31.5,34.3,36.9,,,,,, | |
| 9,9,9,ElevenLabs-FA,ElevenLabs FA,"forced-alignment, API v1",elevenlabs,commercial,forced alignment,API,0.280,0.274,0.287,,0.015,0.22,0.295,0.281,0.309,0.293,0.280,0.306,0.271,0.269,0.274,0.261,0.259,0.264,31.9,32.8,34.0,35.5,,,,,, | |
| 10,10,11,BFA,BFA,,bfa,open,forced alignment,CTC,0.263,0.255,0.271,,0.009,,0.272,0.255,0.290,0.251,0.236,0.265,0.283,0.279,0.286,0.247,0.244,0.250,51.0,56.9,48.6,61.5,,,,,, | |
| 11,10,12,WhisperX,WhisperX,,whisperx,open,forced alignment,CTC,0.257,0.250,0.263,,0.014,,0.276,0.261,0.289,0.263,0.251,0.275,0.259,0.256,0.262,0.230,0.227,0.232,37.3,39.4,39.7,48.2,,,,,, | |
| 12,11,12,TorchAudio,TorchAudio,,pytorch,open,forced alignment,CTC,0.252,0.245,0.260,,0.013,,0.269,0.254,0.285,0.252,0.239,0.265,0.254,0.251,0.257,0.234,0.231,0.236,39.9,42.2,39.8,46.9,,,,,, | |
| 13,13,13,stable-ts,stable-ts,,stablets,open,forced alignment,Attention,0.236,0.228,0.244,,0.015,,0.193,0.175,0.210,0.204,0.189,0.220,0.278,0.274,0.281,0.270,0.267,0.274,97.8,95.3,70.8,73.3,,,,,, | |
| 14,14,14,NVIDIA NeMo-FA,NeMo-FA 80 ms,,nvidia,open,forced alignment,CTC,0.147,0.140,0.154,,0.008,,0.154,0.140,0.169,0.154,0.141,0.168,0.139,0.137,0.142,0.140,0.137,0.142,79.3,79.8,88.8,88.9,,,,,†,NeMo-FA here is its 80 ms model. The 40 ms model's mean F1 is 0.189. | |
| ,,,Olewave Olign 0.9,Olign 0.9,olign-0.9,olewave,commercial,forced alignment,API,0.704,0.694,0.712,,0.031,,0.791,0.771,0.810,0.697,0.677,0.714,0.703,0.699,0.708,0.623,0.619,0.628,18.3,32.8,20.8,29.3,,,,,, | |
| </script> | |
| <script type="text/csv" data-board="fa-phone-recognized" data-track="1" data-tier="phone" data-href="https://olewave.com/media/blog/fa-bench-leaderboard/track1_phone_f1_recognized.csv?v=1793210021" data-words="recognized" data-note="Qwen3-ASR's words have a WER of 2.0% on TIMIT core test and 10.2% on Buckeye test on clean audio. NeuFA was not run on recognized words. Olign v1 was not run on recognized words, so the unranked Olign row here is Olign 0.9."> | |
| rank,rank_lo,rank_hi,label,system,version,org,access,pipeline,family,f1_mean4,f1_mean4_lo,f1_mean4_hi,f1_mean4_recognized,f1_drop,price_usd_per_hour,f1_timit_clean,f1_timit_clean_lo,f1_timit_clean_hi,f1_timit_noisy,f1_timit_noisy_lo,f1_timit_noisy_hi,f1_buckeye_clean,f1_buckeye_clean_lo,f1_buckeye_clean_hi,f1_buckeye_noisy,f1_buckeye_noisy_lo,f1_buckeye_noisy_hi,mae_ms_timit_clean,mae_ms_timit_noisy,mae_ms_buckeye_clean,mae_ms_buckeye_noisy,per_pct_timit_clean,per_pct_timit_noisy,per_pct_buckeye_clean,per_pct_buckeye_noisy,flag,note | |
| 1,1,1,MFA 3.4,MFA 3.4,,mfa,open,forced alignment,HMM,0.428,0.423,0.434,,0.008,,0.394,0.382,0.405,0.372,0.360,0.383,0.502,0.498,0.506,0.446,0.442,0.450,11.6,16.6,17.6,27.2,34.8,35.2,28.2,32.4,, | |
| 2,2,2,PocketSphinx,PocketSphinx,,cmusphinx,open,forced alignment,HMM,0.395,0.389,0.401,,0.007,,0.337,0.325,0.350,0.325,0.314,0.336,0.482,0.479,0.486,0.436,0.433,0.440,16.5,19.8,18.6,27.2,34.0,34.4,27.2,29.6,, | |
| 3,3,3,MAPS,MAPS,,maps,open,forced alignment,Frame,0.378,0.372,0.384,,0.005,,0.455,0.441,0.468,0.319,0.308,0.330,0.435,0.431,0.438,0.303,0.299,0.306,15.2,88.5,26.9,108.9,30.7,30.8,35.8,37.1,⚠,"MAPS was trained on 7 of the 8 speakers in FA-Bench's Buckeye test set, so most of its Buckeye test audio comes from speakers it was trained on." | |
| 4,4,4,Charsiu,Charsiu,,charsiu,open,forced alignment,Frame,0.369,0.363,0.375,,0.003,,0.361,0.349,0.373,0.347,0.336,0.358,0.417,0.413,0.420,0.352,0.349,0.355,19.8,28.0,21.2,45.8,30.3,31.2,32.7,36.7,, | |
| 5,5,5,TorchAudio,TorchAudio,,pytorch,open,forced alignment,CTC,0.230,0.226,0.235,,0.005,,0.207,0.198,0.217,0.205,0.196,0.214,0.260,0.257,0.263,0.250,0.247,0.253,33.8,33.4,34.1,38.9,35.9,36.0,30.0,31.5,, | |
| 6,6,6,BFA,BFA,,bfa,open,forced alignment,CTC,0.213,0.208,0.217,,0.004,,0.183,0.174,0.192,0.177,0.169,0.186,0.253,0.250,0.256,0.238,0.235,0.241,45.1,48.9,44.2,57.7,36.2,36.3,33.6,35.0,, | |
| ,,,Olewave Olign 0.9,Olign 0.9,olign-0.9,olewave,commercial,forced alignment,API,0.447,0.440,0.453,,0.005,,0.410,0.397,0.422,0.393,0.381,0.405,0.513,0.509,0.517,0.470,0.466,0.474,7.7,12.3,12.8,21.8,35.4,35.6,29.2,31.0,, | |
| </script> | |
| <section class="fab-lb__static"> | |
| <h2>ASR with timestamps, word timestamp F1 at 20 ms</h2> | |
| <div class="fab-lb__table-wrap"> | |
| <table class="fab-lb__tbl"> | |
| <thead><tr> | |
| <th class="num" scope="col">Rank</th><th class="num" scope="col">Range</th><th scope="col">System</th><th scope="col">Access</th> | |
| <th class="num" scope="col">Mean F1</th><th class="num" scope="col">95% interval</th><th class="num" scope="col">TIMIT clean</th><th class="num" scope="col">TIMIT noisy</th> | |
| <th class="num" scope="col">Buckeye clean</th><th class="num" scope="col">Buckeye noisy</th> | |
| <th class="num" scope="col">WER TIMIT</th><th class="num" scope="col">WER Buckeye</th> | |
| <th class="num" scope="col">Price, $ per audio hour</th> | |
| </tr></thead> | |
| <tbody> | |
| <tr> | |
| <td class="num">1</td> | |
| <td class="num">1</td> | |
| <td class="fab-lb__sys-cell">Google Chirp 2</td> | |
| <td>Commercial</td> | |
| <td class="num">0.486</td> | |
| <td class="num">0.475–0.498</td> | |
| <td class="num">0.488</td><td class="num">0.497</td><td class="num">0.492</td><td class="num">0.469</td> | |
| <td class="num">4.2</td><td class="num">13.1</td> | |
| <td class="num">$0.96</td> | |
| </tr> | |
| <tr> | |
| <td class="num">2</td> | |
| <td class="num">2–3</td> | |
| <td class="fab-lb__sys-cell">CrisperWhisper ⚠</td> | |
| <td>Open</td> | |
| <td class="num">0.455</td> | |
| <td class="num">0.442–0.466</td> | |
| <td class="num">0.484</td><td class="num">0.472</td><td class="num">0.463</td><td class="num">0.399</td> | |
| <td class="num">3.6</td><td class="num">11.5</td> | |
| <td class="num">–</td> | |
| </tr> | |
| <tr> | |
| <td class="num">3</td> | |
| <td class="num">2–3</td> | |
| <td class="fab-lb__sys-cell">Google Chirp 3 ‡</td> | |
| <td>Commercial</td> | |
| <td class="num">0.452</td> | |
| <td class="num">0.442–0.462</td> | |
| <td class="num">0.460</td><td class="num">0.438</td><td class="num">0.479</td><td class="num">0.431</td> | |
| <td class="num">5.2</td><td class="num">13.5</td> | |
| <td class="num">$0.96</td> | |
| </tr> | |
| <tr> | |
| <td class="num">4</td> | |
| <td class="num">4</td> | |
| <td class="fab-lb__sys-cell">Qwen3-ASR-Flash</td> | |
| <td>Commercial</td> | |
| <td class="num">0.363</td> | |
| <td class="num">0.353–0.374</td> | |
| <td class="num">0.398</td><td class="num">0.355</td><td class="num">0.376</td><td class="num">0.322</td> | |
| <td class="num">1.8</td><td class="num">10.1</td> | |
| <td class="num">$0.13</td> | |
| </tr> | |
| <tr> | |
| <td class="num">5</td> | |
| <td class="num">5</td> | |
| <td class="fab-lb__sys-cell">Mistral Voxtral Mini</td> | |
| <td>Commercial</td> | |
| <td class="num">0.308</td> | |
| <td class="num">0.298–0.319</td> | |
| <td class="num">0.315</td><td class="num">0.294</td><td class="num">0.325</td><td class="num">0.299</td> | |
| <td class="num">2.4</td><td class="num">13.4</td> | |
| <td class="num">$0.18</td> | |
| </tr> | |
| <tr> | |
| <td class="num">6</td> | |
| <td class="num">6–7</td> | |
| <td class="fab-lb__sys-cell">ElevenLabs Scribe v2</td> | |
| <td>Commercial</td> | |
| <td class="num">0.281</td> | |
| <td class="num">0.275–0.288</td> | |
| <td class="num">0.304</td><td class="num">0.297</td><td class="num">0.267</td><td class="num">0.256</td> | |
| <td class="num">2.1</td><td class="num">10.7</td> | |
| <td class="num">$0.22</td> | |
| </tr> | |
| <tr> | |
| <td class="num">7</td> | |
| <td class="num">6–7</td> | |
| <td class="fab-lb__sys-cell">Amazon Transcribe</td> | |
| <td>Commercial</td> | |
| <td class="num">0.273</td> | |
| <td class="num">0.263–0.284</td> | |
| <td class="num">0.259</td><td class="num">0.264</td><td class="num">0.289</td><td class="num">0.281</td> | |
| <td class="num">2.7</td><td class="num">10.3</td> | |
| <td class="num">$0.36</td> | |
| </tr> | |
| <tr> | |
| <td class="num">8</td> | |
| <td class="num">8–9</td> | |
| <td class="fab-lb__sys-cell">AssemblyAI Universal-3.5 Pro</td> | |
| <td>Commercial</td> | |
| <td class="num">0.213</td> | |
| <td class="num">0.204–0.222</td> | |
| <td class="num">0.237</td><td class="num">0.218</td><td class="num">0.211</td><td class="num">0.185</td> | |
| <td class="num">2.0</td><td class="num">9.0</td> | |
| <td class="num">$0.21</td> | |
| </tr> | |
| <tr> | |
| <td class="num">9</td> | |
| <td class="num">8–9</td> | |
| <td class="fab-lb__sys-cell">TorchAudio (ASR)</td> | |
| <td>Open</td> | |
| <td class="num">0.209</td> | |
| <td class="num">0.203–0.216</td> | |
| <td class="num">0.254</td><td class="num">0.213</td><td class="num">0.211</td><td class="num">0.159</td> | |
| <td class="num">10.8</td><td class="num">27.8</td> | |
| <td class="num">–</td> | |
| </tr> | |
| <tr> | |
| <td class="num">10</td> | |
| <td class="num">10–11</td> | |
| <td class="fab-lb__sys-cell">Deepgram Nova-3 ‡</td> | |
| <td>Commercial</td> | |
| <td class="num">0.170</td> | |
| <td class="num">0.161–0.178</td> | |
| <td class="num">0.191</td><td class="num">0.185</td><td class="num">0.155</td><td class="num">0.148</td> | |
| <td class="num">2.4</td><td class="num">10.8</td> | |
| <td class="num">$0.26</td> | |
| </tr> | |
| <tr> | |
| <td class="num">11</td> | |
| <td class="num">10–11</td> | |
| <td class="fab-lb__sys-cell">Parakeet-TDT</td> | |
| <td>Open</td> | |
| <td class="num">0.164</td> | |
| <td class="num">0.155–0.172</td> | |
| <td class="num">0.172</td><td class="num">0.174</td><td class="num">0.156</td><td class="num">0.153</td> | |
| <td class="num">2.4</td><td class="num">11.0</td> | |
| <td class="num">–</td> | |
| </tr> | |
| <tr> | |
| <td class="num">12</td> | |
| <td class="num">12</td> | |
| <td class="fab-lb__sys-cell">IBM Watson STT en-US</td> | |
| <td>Commercial</td> | |
| <td class="num">0.149</td> | |
| <td class="num">0.142–0.157</td> | |
| <td class="num">0.135</td><td class="num">0.132</td><td class="num">0.176</td><td class="num">0.155</td> | |
| <td class="num">5.1</td><td class="num">12.4</td> | |
| <td class="num">$1.27</td> | |
| </tr> | |
| <tr> | |
| <td class="num">13</td> | |
| <td class="num">13–14</td> | |
| <td class="fab-lb__sys-cell">Azure AI Speech</td> | |
| <td>Commercial</td> | |
| <td class="num">0.100</td> | |
| <td class="num">0.094–0.107</td> | |
| <td class="num">0.117</td><td class="num">0.112</td><td class="num">0.089</td><td class="num">0.083</td> | |
| <td class="num">3.4</td><td class="num">11.5</td> | |
| <td class="num">$1.00</td> | |
| </tr> | |
| <tr> | |
| <td class="num">14</td> | |
| <td class="num">13–14</td> | |
| <td class="fab-lb__sys-cell">Speechmatics Enhanced</td> | |
| <td>Commercial</td> | |
| <td class="num">0.097</td> | |
| <td class="num">0.091–0.103</td> | |
| <td class="num">0.110</td><td class="num">0.088</td><td class="num">0.107</td><td class="num">0.083</td> | |
| <td class="num">2.5</td><td class="num">11.4</td> | |
| <td class="num">$0.75</td> | |
| </tr> | |
| <tr> | |
| <td class="num">15</td> | |
| <td class="num">15–16</td> | |
| <td class="fab-lb__sys-cell">Whisper large-v3</td> | |
| <td>Open</td> | |
| <td class="num">0.049</td> | |
| <td class="num">0.046–0.052</td> | |
| <td class="num">0.029</td><td class="num">0.029</td><td class="num">0.067</td><td class="num">0.069</td> | |
| <td class="num">2.9</td><td class="num">13.7</td> | |
| <td class="num">–</td> | |
| </tr> | |
| <tr> | |
| <td class="num">16</td> | |
| <td class="num">15–16</td> | |
| <td class="fab-lb__sys-cell">Whisper-timestamped</td> | |
| <td>Open</td> | |
| <td class="num">0.049</td> | |
| <td class="num">0.046–0.052</td> | |
| <td class="num">0.021</td><td class="num">0.022</td><td class="num">0.075</td><td class="num">0.076</td> | |
| <td class="num">2.5</td><td class="num">13.4</td> | |
| <td class="num">–</td> | |
| </tr> | |
| <tr class="is-withdrawn"> | |
| <td class="num">–</td> | |
| <td class="num">–</td> | |
| <td class="fab-lb__sys-cell">Olewave <a href="https://olewave.com/en/olign-olewaves-lancet-accurate-speech-to-text-forced-alignment-service/">Olign</a> v1</td> | |
| <td>Commercial</td> | |
| <td class="num">0.681</td> | |
| <td class="num">0.673–0.690</td> | |
| <td class="num">0.786</td><td class="num">0.695</td><td class="num">0.660</td><td class="num">0.585</td> | |
| <td class="num">2.2</td><td class="num">10.2</td> | |
| <td class="num">$0.18</td> | |
| </tr> | |
| </tbody> | |
| </table> | |
| </div> | |
| <p class="fab-lb__notes">⚠ CrisperWhisper's authors used TIMIT to build its word timing and don't say which TIMIT files. <a href="https://github.com/olewave/fa-bench/">FA-Bench</a> tests on TIMIT's test files, so if the authors used those too, its TIMIT scores come from audio it has already seen.</p><p class="fab-lb__notes">‡ Chirp 3 ends the last word of 15% of clean Buckeye test utterances more than 0.3 s after the audio does.</p><p class="fab-lb__notes">‡ Deepgram ends the last word of 24% of clean Buckeye test utterances more than 0.3 s after the audio does.</p><p class="fab-lb__notes">Every row is one system as shipped. A cascade of a recognizer and a forced aligner is not ranked here. It shows on the forced alignment boards under Recognized words, and in the F1 Drop of its aligner's row. <a href="https://olewave.com/en/olign-olewaves-lancet-accurate-speech-to-text-forced-alignment-service/">Olign</a>'s price is $0.18 an hour, which is its launch promotion price. Its list price is $0.21.</p><p class="fab-lb__notes">Google Chirp 2 is chirp_2, Speech-to-Text API v2. Google Chirp 3 is chirp_3, Speech-to-Text API v2. Qwen3-ASR-Flash is qwen3-asr-flash-filetrans, DashScope API v1. Mistral Voxtral Mini is voxtral-mini-2602, API v1. ElevenLabs Scribe v2 is scribe_v2, API v1. Amazon Transcribe is en-US, batch API 2017-10-26. AssemblyAI Universal-3.5 Pro is universal-3-5-pro, API v2. Deepgram Nova-3 is nova-3, build 2025-07-31.0, API v1. IBM Watson STT en-US is en-US large speech model, API v1. Azure AI Speech is en-US, short audio REST API v1. Speechmatics Enhanced is enhanced, builds 16.0.0 and 16.1.0, API v2. Olewave <a href="https://olewave.com/en/olign-olewaves-lancet-accurate-speech-to-text-forced-alignment-service/">Olign</a> v1 is olign-1.0, <a href="https://olewave.com/en/olign-olewaves-lancet-accurate-speech-to-text-forced-alignment-service/">Olign</a> API v1.</p><p class="fab-lb__notes"><a href="https://olewave.com/en/olign-olewaves-lancet-accurate-speech-to-text-forced-alignment-service/">Olign</a> no longer takes part in <a href="https://github.com/olewave/fa-bench/">FA-Bench</a>. Its last results, from October 2026, are listed unranked at the bottom of the table.</p> | |
| <p class="fab-lb__source"><a href="https://olewave.com/media/blog/fa-bench-leaderboard/track2_word_f1.csv?v=1793210021" download>Download the CSV</a>.</p> | |
| </section> | |
| <section class="fab-lb__static"> | |
| <h2>Forced alignment, word timestamp F1 at 20 ms</h2> | |
| <div class="fab-lb__table-wrap"> | |
| <table class="fab-lb__tbl"> | |
| <thead><tr> | |
| <th class="num" scope="col">Rank</th><th class="num" scope="col">Range</th><th scope="col">System</th><th scope="col">Access</th> | |
| <th class="num" scope="col">Mean F1</th><th class="num" scope="col">95% interval</th><th class="num" scope="col">F1 Drop</th><th class="num" scope="col">TIMIT clean</th><th class="num" scope="col">TIMIT noisy</th> | |
| <th class="num" scope="col">Buckeye clean</th><th class="num" scope="col">Buckeye noisy</th> | |
| <th class="num" scope="col">MAE TIMIT</th><th class="num" scope="col">MAE Buckeye</th> | |
| <th class="num" scope="col">Price, $ per audio hour</th> | |
| </tr></thead> | |
| <tbody> | |
| <tr> | |
| <td class="num">1</td> | |
| <td class="num">1</td> | |
| <td class="fab-lb__sys-cell">MFA 3.4</td> | |
| <td>Open</td> | |
| <td class="num">0.658</td> | |
| <td class="num">0.650–0.667</td> | |
| <td class="num">0.030</td> | |
| <td class="num">0.685</td><td class="num">0.649</td><td class="num">0.706</td><td class="num">0.594</td> | |
| <td class="num">21.8</td><td class="num">21.3</td> | |
| <td class="num">–</td> | |
| </tr> | |
| <tr> | |
| <td class="num">2</td> | |
| <td class="num">2</td> | |
| <td class="fab-lb__sys-cell">PocketSphinx</td> | |
| <td>Open</td> | |
| <td class="num">0.648</td> | |
| <td class="num">0.639–0.658</td> | |
| <td class="num">0.033</td> | |
| <td class="num">0.645</td><td class="num">0.594</td><td class="num">0.728</td><td class="num">0.627</td> | |
| <td class="num">30.5</td><td class="num">22.1</td> | |
| <td class="num">–</td> | |
| </tr> | |
| <tr> | |
| <td class="num">3</td> | |
| <td class="num">3</td> | |
| <td class="fab-lb__sys-cell">NeuFA ⚠</td> | |
| <td>Open</td> | |
| <td class="num">0.599</td> | |
| <td class="num">0.589–0.607</td> | |
| <td class="num">–</td> | |
| <td class="num">0.600</td><td class="num">0.493</td><td class="num">0.712</td><td class="num">0.589</td> | |
| <td class="num">56.9</td><td class="num">33.9</td> | |
| <td class="num">–</td> | |
| </tr> | |
| <tr> | |
| <td class="num">4</td> | |
| <td class="num">4</td> | |
| <td class="fab-lb__sys-cell">Charsiu</td> | |
| <td>Open</td> | |
| <td class="num">0.563</td> | |
| <td class="num">0.553–0.573</td> | |
| <td class="num">0.024</td> | |
| <td class="num">0.597</td><td class="num">0.532</td><td class="num">0.627</td><td class="num">0.497</td> | |
| <td class="num">28.1</td><td class="num">28.1</td> | |
| <td class="num">–</td> | |
| </tr> | |
| <tr> | |
| <td class="num">5</td> | |
| <td class="num">5</td> | |
| <td class="fab-lb__sys-cell">MAPS ⚠</td> | |
| <td>Open</td> | |
| <td class="num">0.547</td> | |
| <td class="num">0.537–0.556</td> | |
| <td class="num">0.024</td> | |
| <td class="num">0.712</td><td class="num">0.412</td><td class="num">0.642</td><td class="num">0.421</td> | |
| <td class="num">25.5</td><td class="num">37.9</td> | |
| <td class="num">–</td> | |
| </tr> | |
| <tr> | |
| <td class="num">6</td> | |
| <td class="num">6</td> | |
| <td class="fab-lb__sys-cell">CrisperWhisper ⚠</td> | |
| <td>Open</td> | |
| <td class="num">0.473</td> | |
| <td class="num">0.461–0.485</td> | |
| <td class="num">0.017</td> | |
| <td class="num">0.489</td><td class="num">0.480</td><td class="num">0.491</td><td class="num">0.434</td> | |
| <td class="num">34.1</td><td class="num">43.1</td> | |
| <td class="num">–</td> | |
| </tr> | |
| <tr> | |
| <td class="num">7</td> | |
| <td class="num">7</td> | |
| <td class="fab-lb__sys-cell">Meta UnitY2</td> | |
| <td>Open</td> | |
| <td class="num">0.439</td> | |
| <td class="num">0.428–0.449</td> | |
| <td class="num">0.022</td> | |
| <td class="num">0.380</td><td class="num">0.393</td><td class="num">0.495</td><td class="num">0.486</td> | |
| <td class="num">52.2</td><td class="num">40.6</td> | |
| <td class="num">–</td> | |
| </tr> | |
| <tr> | |
| <td class="num">8</td> | |
| <td class="num">8</td> | |
| <td class="fab-lb__sys-cell">Qwen3-FA</td> | |
| <td>Open</td> | |
| <td class="num">0.402</td> | |
| <td class="num">0.391–0.413</td> | |
| <td class="num">0.013</td> | |
| <td class="num">0.416</td><td class="num">0.386</td><td class="num">0.426</td><td class="num">0.380</td> | |
| <td class="num">32.2</td><td class="num">33.8</td> | |
| <td class="num">–</td> | |
| </tr> | |
| <tr> | |
| <td class="num">9</td> | |
| <td class="num">9</td> | |
| <td class="fab-lb__sys-cell">Meta MMS-FA</td> | |
| <td>Open</td> | |
| <td class="num">0.345</td> | |
| <td class="num">0.338–0.352</td> | |
| <td class="num">0.014</td> | |
| <td class="num">0.353</td><td class="num">0.350</td><td class="num">0.344</td><td class="num">0.332</td> | |
| <td class="num">31.1</td><td class="num">37.1</td> | |
| <td class="num">–</td> | |
| </tr> | |
| <tr> | |
| <td class="num">10</td> | |
| <td class="num">10</td> | |
| <td class="fab-lb__sys-cell">ElevenLabs-FA</td> | |
| <td>Commercial</td> | |
| <td class="num">0.295</td> | |
| <td class="num">0.289–0.302</td> | |
| <td class="num">0.015</td> | |
| <td class="num">0.299</td><td class="num">0.300</td><td class="num">0.293</td><td class="num">0.288</td> | |
| <td class="num">31.9</td><td class="num">35.5</td> | |
| <td class="num">$0.22</td> | |
| </tr> | |
| <tr> | |
| <td class="num">11</td> | |
| <td class="num">11–13</td> | |
| <td class="fab-lb__sys-cell">BFA</td> | |
| <td>Open</td> | |
| <td class="num">0.272</td> | |
| <td class="num">0.264–0.280</td> | |
| <td class="num">0.009</td> | |
| <td class="num">0.276</td><td class="num">0.255</td><td class="num">0.285</td><td class="num">0.272</td> | |
| <td class="num">51.0</td><td class="num">57.2</td> | |
| <td class="num">–</td> | |
| </tr> | |
| <tr> | |
| <td class="num">12</td> | |
| <td class="num">11–13</td> | |
| <td class="fab-lb__sys-cell">WhisperX</td> | |
| <td>Open</td> | |
| <td class="num">0.271</td> | |
| <td class="num">0.264–0.277</td> | |
| <td class="num">0.014</td> | |
| <td class="num">0.280</td><td class="num">0.267</td><td class="num">0.281</td><td class="num">0.254</td> | |
| <td class="num">37.2</td><td class="num">41.7</td> | |
| <td class="num">–</td> | |
| </tr> | |
| <tr> | |
| <td class="num">13</td> | |
| <td class="num">12–13</td> | |
| <td class="fab-lb__sys-cell">TorchAudio</td> | |
| <td>Open</td> | |
| <td class="num">0.265</td> | |
| <td class="num">0.258–0.272</td> | |
| <td class="num">0.013</td> | |
| <td class="num">0.273</td><td class="num">0.255</td><td class="num">0.275</td><td class="num">0.256</td> | |
| <td class="num">39.8</td><td class="num">42.4</td> | |
| <td class="num">–</td> | |
| </tr> | |
| <tr> | |
| <td class="num">14</td> | |
| <td class="num">14</td> | |
| <td class="fab-lb__sys-cell">stable-ts</td> | |
| <td>Open</td> | |
| <td class="num">0.251</td> | |
| <td class="num">0.243–0.260</td> | |
| <td class="num">0.015</td> | |
| <td class="num">0.197</td><td class="num">0.210</td><td class="num">0.301</td><td class="num">0.297</td> | |
| <td class="num">96.8</td><td class="num">71.2</td> | |
| <td class="num">–</td> | |
| </tr> | |
| <tr> | |
| <td class="num">15</td> | |
| <td class="num">15</td> | |
| <td class="fab-lb__sys-cell">NVIDIA NeMo-FA †</td> | |
| <td>Open</td> | |
| <td class="num">0.155</td> | |
| <td class="num">0.149–0.163</td> | |
| <td class="num">0.008</td> | |
| <td class="num">0.157</td><td class="num">0.158</td><td class="num">0.151</td><td class="num">0.156</td> | |
| <td class="num">79.2</td><td class="num">91.4</td> | |
| <td class="num">–</td> | |
| </tr> | |
| <tr class="is-withdrawn"> | |
| <td class="num">–</td> | |
| <td class="num">–</td> | |
| <td class="fab-lb__sys-cell">Olewave <a href="https://olewave.com/en/olign-olewaves-lancet-accurate-speech-to-text-forced-alignment-service/">Olign</a> v1</td> | |
| <td>Commercial</td> | |
| <td class="num">0.735</td> | |
| <td class="num">0.726–0.743</td> | |
| <td class="num">–</td> | |
| <td class="num">0.805</td><td class="num">0.710</td><td class="num">0.755</td><td class="num">0.668</td> | |
| <td class="num">18.2</td><td class="num">22.4</td> | |
| <td class="num">$0.18</td> | |
| </tr> | |
| </tbody> | |
| </table> | |
| </div> | |
| <p class="fab-lb__drop">F1 Drop is how far an aligner's mean F1 falls when it is handed the words Qwen3-ASR recognized in place of the reference transcript. MFA 3.4, for example, scores 0.658 on the reference transcript and 0.628 on recognized words, an F1 Drop of 0.030.</p> | |
| <p class="fab-lb__notes">⚠ NeuFA was trained on 7 of the 8 speakers in <a href="https://github.com/olewave/fa-bench/">FA-Bench</a>'s Buckeye test set, so most of its Buckeye test audio comes from speakers it was trained on.</p><p class="fab-lb__notes">⚠ MAPS was trained on 7 of the 8 speakers in <a href="https://github.com/olewave/fa-bench/">FA-Bench</a>'s Buckeye test set, so most of its Buckeye test audio comes from speakers it was trained on.</p><p class="fab-lb__notes">⚠ CrisperWhisper's authors used TIMIT to build its word timing and don't say which TIMIT files. <a href="https://github.com/olewave/fa-bench/">FA-Bench</a> tests on TIMIT's test files, so if the authors used those too, its TIMIT scores come from audio it has already seen.</p><p class="fab-lb__notes">† NeMo-FA here is its 80 ms model. The 40 ms model's mean F1 is 0.199.</p><p class="fab-lb__notes">The Transcript switch shows each aligner handed the words Qwen3-ASR recognized in place of the reference transcript. Those words have a WER of 2.0% on TIMIT core test and 10.2% on Buckeye test on clean audio. NeuFA was not run on recognized words. MFA 3.4 on Parakeet-TDT's words scores 0.625, and WhisperX on Whisper large-v3's words scores 0.234. <a href="https://olewave.com/en/olign-olewaves-lancet-accurate-speech-to-text-forced-alignment-service/">Olign</a>'s price is $0.18 an hour, which is its launch promotion price. Its list price is $0.21.</p><p class="fab-lb__notes">NeuFA is trained with its open-source code on GitHub, not released. ElevenLabs-FA is forced-alignment, API v1. Olewave <a href="https://olewave.com/en/olign-olewaves-lancet-accurate-speech-to-text-forced-alignment-service/">Olign</a> v1 is olign-1.0, <a href="https://olewave.com/en/olign-olewaves-lancet-accurate-speech-to-text-forced-alignment-service/">Olign</a> API v1.</p><p class="fab-lb__notes"><a href="https://olewave.com/en/olign-olewaves-lancet-accurate-speech-to-text-forced-alignment-service/">Olign</a> no longer takes part in <a href="https://github.com/olewave/fa-bench/">FA-Bench</a>. Its last results, from October 2026, are listed unranked at the bottom of the table.</p> | |
| <p class="fab-lb__source"><a href="https://olewave.com/media/blog/fa-bench-leaderboard/track1_word_f1.csv?v=1793210021" download>Download the CSV</a>.</p> | |
| </section> | |
| <section class="fab-lb__static"> | |
| <h2>Forced alignment, phone boundary F1 at 20 ms</h2> | |
| <div class="fab-lb__table-wrap"> | |
| <table class="fab-lb__tbl"> | |
| <thead><tr> | |
| <th class="num" scope="col">Rank</th><th class="num" scope="col">Range</th><th scope="col">System</th><th scope="col">Access</th> | |
| <th class="num" scope="col">Mean F1</th><th class="num" scope="col">95% interval</th><th class="num" scope="col">F1 Drop</th><th class="num" scope="col">TIMIT clean</th><th class="num" scope="col">TIMIT noisy</th> | |
| <th class="num" scope="col">Buckeye clean</th><th class="num" scope="col">Buckeye noisy</th> | |
| <th class="num" scope="col">PER TIMIT</th><th class="num" scope="col">PER Buckeye</th> | |
| <th class="num" scope="col">Price, $ per audio hour</th> | |
| </tr></thead> | |
| <tbody> | |
| <tr> | |
| <td class="num">1</td> | |
| <td class="num">1</td> | |
| <td class="fab-lb__sys-cell">MFA 3.4</td> | |
| <td>Open</td> | |
| <td class="num">0.436</td> | |
| <td class="num">0.431–0.443</td> | |
| <td class="num">0.008</td> | |
| <td class="num">0.393</td><td class="num">0.372</td><td class="num">0.518</td><td class="num">0.463</td> | |
| <td class="num">34.8</td><td class="num">26.1</td> | |
| <td class="num">–</td> | |
| </tr> | |
| <tr> | |
| <td class="num">2</td> | |
| <td class="num">2</td> | |
| <td class="fab-lb__sys-cell">PocketSphinx</td> | |
| <td>Open</td> | |
| <td class="num">0.402</td> | |
| <td class="num">0.396–0.408</td> | |
| <td class="num">0.007</td> | |
| <td class="num">0.338</td><td class="num">0.326</td><td class="num">0.494</td><td class="num">0.450</td> | |
| <td class="num">33.9</td><td class="num">26.0</td> | |
| <td class="num">–</td> | |
| </tr> | |
| <tr> | |
| <td class="num">3</td> | |
| <td class="num">3</td> | |
| <td class="fab-lb__sys-cell">MAPS ⚠</td> | |
| <td>Open</td> | |
| <td class="num">0.383</td> | |
| <td class="num">0.377–0.389</td> | |
| <td class="num">0.005</td> | |
| <td class="num">0.454</td><td class="num">0.319</td><td class="num">0.445</td><td class="num">0.313</td> | |
| <td class="num">30.7</td><td class="num">34.9</td> | |
| <td class="num">–</td> | |
| </tr> | |
| <tr> | |
| <td class="num">4</td> | |
| <td class="num">4</td> | |
| <td class="fab-lb__sys-cell">Charsiu</td> | |
| <td>Open</td> | |
| <td class="num">0.372</td> | |
| <td class="num">0.367–0.378</td> | |
| <td class="num">0.003</td> | |
| <td class="num">0.360</td><td class="num">0.347</td><td class="num">0.424</td><td class="num">0.359</td> | |
| <td class="num">30.3</td><td class="num">31.7</td> | |
| <td class="num">–</td> | |
| </tr> | |
| <tr> | |
| <td class="num">5</td> | |
| <td class="num">5</td> | |
| <td class="fab-lb__sys-cell">NeuFA ⚠</td> | |
| <td>Open</td> | |
| <td class="num">0.366</td> | |
| <td class="num">0.360–0.371</td> | |
| <td class="num">–</td> | |
| <td class="num">0.326</td><td class="num">0.277</td><td class="num">0.460</td><td class="num">0.399</td> | |
| <td class="num">36.0</td><td class="num">28.0</td> | |
| <td class="num">–</td> | |
| </tr> | |
| <tr> | |
| <td class="num">6</td> | |
| <td class="num">6</td> | |
| <td class="fab-lb__sys-cell">TorchAudio</td> | |
| <td>Open</td> | |
| <td class="num">0.235</td> | |
| <td class="num">0.230–0.240</td> | |
| <td class="num">0.005</td> | |
| <td class="num">0.208</td><td class="num">0.206</td><td class="num">0.266</td><td class="num">0.261</td> | |
| <td class="num">35.9</td><td class="num">28.5</td> | |
| <td class="num">–</td> | |
| </tr> | |
| <tr> | |
| <td class="num">7</td> | |
| <td class="num">7</td> | |
| <td class="fab-lb__sys-cell">BFA</td> | |
| <td>Open</td> | |
| <td class="num">0.217</td> | |
| <td class="num">0.212–0.221</td> | |
| <td class="num">0.004</td> | |
| <td class="num">0.183</td><td class="num">0.178</td><td class="num">0.255</td><td class="num">0.251</td> | |
| <td class="num">36.2</td><td class="num">32.1</td> | |
| <td class="num">–</td> | |
| </tr> | |
| <tr class="is-withdrawn"> | |
| <td class="num">–</td> | |
| <td class="num">–</td> | |
| <td class="fab-lb__sys-cell">Olewave <a href="https://olewave.com/en/olign-olewaves-lancet-accurate-speech-to-text-forced-alignment-service/">Olign</a> v1</td> | |
| <td>Commercial</td> | |
| <td class="num">0.452</td> | |
| <td class="num">0.446–0.459</td> | |
| <td class="num">–</td> | |
| <td class="num">0.410</td><td class="num">0.393</td><td class="num">0.523</td><td class="num">0.483</td> | |
| <td class="num">35.4</td><td class="num">28.2</td> | |
| <td class="num">$0.18</td> | |
| </tr> | |
| </tbody> | |
| </table> | |
| </div> | |
| <p class="fab-lb__drop">F1 Drop is how far an aligner's mean F1 falls when it is handed the words Qwen3-ASR recognized in place of the reference transcript. MFA 3.4, for example, scores 0.436 on the reference transcript and 0.428 on recognized words, an F1 Drop of 0.008.</p> | |
| <p class="fab-lb__notes">⚠ MAPS was trained on 7 of the 8 speakers in <a href="https://github.com/olewave/fa-bench/">FA-Bench</a>'s Buckeye test set, so most of its Buckeye test audio comes from speakers it was trained on.</p><p class="fab-lb__notes">⚠ NeuFA was trained on 7 of the 8 speakers in <a href="https://github.com/olewave/fa-bench/">FA-Bench</a>'s Buckeye test set, so most of its Buckeye test audio comes from speakers it was trained on.</p><p class="fab-lb__notes">FALCON is left off this board because it is handed the reference phones, which no other aligner here gets. Its mean F1 is 0.560. The Transcript switch shows each aligner handed the words Qwen3-ASR recognized in place of the reference transcript. Those words have a WER of 2.0% on TIMIT core test and 10.2% on Buckeye test on clean audio. NeuFA was not run on recognized words. MFA 3.4 on Parakeet-TDT's words scores 0.429.</p><p class="fab-lb__notes">NeuFA is trained with its open-source code on GitHub, not released. Olewave <a href="https://olewave.com/en/olign-olewaves-lancet-accurate-speech-to-text-forced-alignment-service/">Olign</a> v1 is olign-1.0, <a href="https://olewave.com/en/olign-olewaves-lancet-accurate-speech-to-text-forced-alignment-service/">Olign</a> API v1.</p><p class="fab-lb__notes"><a href="https://olewave.com/en/olign-olewaves-lancet-accurate-speech-to-text-forced-alignment-service/">Olign</a> no longer takes part in <a href="https://github.com/olewave/fa-bench/">FA-Bench</a>. Its last results, from October 2026, are listed unranked at the bottom of the table.</p> | |
| <p class="fab-lb__source"><a href="https://olewave.com/media/blog/fa-bench-leaderboard/track1_phone_f1.csv?v=1793210021" download>Download the CSV</a>.</p> | |
| </section> | |
| <section class="fab-lb__static"> | |
| <h2>Forced alignment with recognized words, word timestamp F1 at 20 ms</h2> | |
| <div class="fab-lb__table-wrap"> | |
| <table class="fab-lb__tbl"> | |
| <thead><tr> | |
| <th class="num" scope="col">Rank</th><th class="num" scope="col">Range</th><th scope="col">System</th><th scope="col">Access</th> | |
| <th class="num" scope="col">Mean F1</th><th class="num" scope="col">95% interval</th><th class="num" scope="col">F1 Drop</th><th class="num" scope="col">TIMIT clean</th><th class="num" scope="col">TIMIT noisy</th> | |
| <th class="num" scope="col">Buckeye clean</th><th class="num" scope="col">Buckeye noisy</th> | |
| <th class="num" scope="col">MAE TIMIT</th><th class="num" scope="col">MAE Buckeye</th> | |
| <th class="num" scope="col">Price, $ per audio hour</th> | |
| </tr></thead> | |
| <tbody> | |
| <tr> | |
| <td class="num">1</td> | |
| <td class="num">1</td> | |
| <td class="fab-lb__sys-cell">MFA 3.4</td> | |
| <td>Open</td> | |
| <td class="num">0.628</td> | |
| <td class="num">0.619–0.637</td> | |
| <td class="num">0.030</td> | |
| <td class="num">0.677</td><td class="num">0.642</td><td class="num">0.649</td><td class="num">0.545</td> | |
| <td class="num">21.8</td><td class="num">22.7</td> | |
| <td class="num">–</td> | |
| </tr> | |
| <tr> | |
| <td class="num">2</td> | |
| <td class="num">2</td> | |
| <td class="fab-lb__sys-cell">PocketSphinx</td> | |
| <td>Open</td> | |
| <td class="num">0.615</td> | |
| <td class="num">0.605–0.624</td> | |
| <td class="num">0.033</td> | |
| <td class="num">0.632</td><td class="num">0.585</td><td class="num">0.669</td><td class="num">0.573</td> | |
| <td class="num">30.6</td><td class="num">22.6</td> | |
| <td class="num">–</td> | |
| </tr> | |
| <tr> | |
| <td class="num">3</td> | |
| <td class="num">3</td> | |
| <td class="fab-lb__sys-cell">Charsiu</td> | |
| <td>Open</td> | |
| <td class="num">0.539</td> | |
| <td class="num">0.529–0.550</td> | |
| <td class="num">0.024</td> | |
| <td class="num">0.587</td><td class="num">0.522</td><td class="num">0.586</td><td class="num">0.463</td> | |
| <td class="num">28.2</td><td class="num">26.1</td> | |
| <td class="num">–</td> | |
| </tr> | |
| <tr> | |
| <td class="num">4</td> | |
| <td class="num">4</td> | |
| <td class="fab-lb__sys-cell">MAPS ⚠</td> | |
| <td>Open</td> | |
| <td class="num">0.523</td> | |
| <td class="num">0.514–0.532</td> | |
| <td class="num">0.024</td> | |
| <td class="num">0.702</td><td class="num">0.407</td><td class="num">0.595</td><td class="num">0.387</td> | |
| <td class="num">25.5</td><td class="num">39.8</td> | |
| <td class="num">–</td> | |
| </tr> | |
| <tr> | |
| <td class="num">5</td> | |
| <td class="num">5</td> | |
| <td class="fab-lb__sys-cell">CrisperWhisper ⚠</td> | |
| <td>Open</td> | |
| <td class="num">0.456</td> | |
| <td class="num">0.443–0.468</td> | |
| <td class="num">0.017</td> | |
| <td class="num">0.485</td><td class="num">0.472</td><td class="num">0.464</td><td class="num">0.403</td> | |
| <td class="num">33.8</td><td class="num">39.2</td> | |
| <td class="num">–</td> | |
| </tr> | |
| <tr> | |
| <td class="num">6</td> | |
| <td class="num">6</td> | |
| <td class="fab-lb__sys-cell">Meta UnitY2</td> | |
| <td>Open</td> | |
| <td class="num">0.417</td> | |
| <td class="num">0.406–0.428</td> | |
| <td class="num">0.022</td> | |
| <td class="num">0.375</td><td class="num">0.385</td><td class="num">0.461</td><td class="num">0.447</td> | |
| <td class="num">52.4</td><td class="num">39.8</td> | |
| <td class="num">–</td> | |
| </tr> | |
| <tr> | |
| <td class="num">7</td> | |
| <td class="num">7</td> | |
| <td class="fab-lb__sys-cell">Qwen3-FA</td> | |
| <td>Open</td> | |
| <td class="num">0.389</td> | |
| <td class="num">0.377–0.399</td> | |
| <td class="num">0.013</td> | |
| <td class="num">0.408</td><td class="num">0.383</td><td class="num">0.405</td><td class="num">0.358</td> | |
| <td class="num">32.6</td><td class="num">30.6</td> | |
| <td class="num">–</td> | |
| </tr> | |
| <tr> | |
| <td class="num">8</td> | |
| <td class="num">8</td> | |
| <td class="fab-lb__sys-cell">Meta MMS-FA</td> | |
| <td>Open</td> | |
| <td class="num">0.331</td> | |
| <td class="num">0.324–0.337</td> | |
| <td class="num">0.014</td> | |
| <td class="num">0.349</td><td class="num">0.346</td><td class="num">0.323</td><td class="num">0.305</td> | |
| <td class="num">31.1</td><td class="num">34.3</td> | |
| <td class="num">–</td> | |
| </tr> | |
| <tr> | |
| <td class="num">9</td> | |
| <td class="num">9</td> | |
| <td class="fab-lb__sys-cell">ElevenLabs-FA</td> | |
| <td>Commercial</td> | |
| <td class="num">0.280</td> | |
| <td class="num">0.274–0.287</td> | |
| <td class="num">0.015</td> | |
| <td class="num">0.295</td><td class="num">0.293</td><td class="num">0.271</td><td class="num">0.261</td> | |
| <td class="num">31.9</td><td class="num">34.0</td> | |
| <td class="num">$0.22</td> | |
| </tr> | |
| <tr> | |
| <td class="num">10</td> | |
| <td class="num">10–11</td> | |
| <td class="fab-lb__sys-cell">BFA</td> | |
| <td>Open</td> | |
| <td class="num">0.263</td> | |
| <td class="num">0.255–0.271</td> | |
| <td class="num">0.009</td> | |
| <td class="num">0.272</td><td class="num">0.251</td><td class="num">0.283</td><td class="num">0.247</td> | |
| <td class="num">51.0</td><td class="num">48.6</td> | |
| <td class="num">–</td> | |
| </tr> | |
| <tr> | |
| <td class="num">11</td> | |
| <td class="num">10–12</td> | |
| <td class="fab-lb__sys-cell">WhisperX</td> | |
| <td>Open</td> | |
| <td class="num">0.257</td> | |
| <td class="num">0.250–0.263</td> | |
| <td class="num">0.014</td> | |
| <td class="num">0.276</td><td class="num">0.263</td><td class="num">0.259</td><td class="num">0.230</td> | |
| <td class="num">37.3</td><td class="num">39.7</td> | |
| <td class="num">–</td> | |
| </tr> | |
| <tr> | |
| <td class="num">12</td> | |
| <td class="num">11–12</td> | |
| <td class="fab-lb__sys-cell">TorchAudio</td> | |
| <td>Open</td> | |
| <td class="num">0.252</td> | |
| <td class="num">0.245–0.260</td> | |
| <td class="num">0.013</td> | |
| <td class="num">0.269</td><td class="num">0.252</td><td class="num">0.254</td><td class="num">0.234</td> | |
| <td class="num">39.9</td><td class="num">39.8</td> | |
| <td class="num">–</td> | |
| </tr> | |
| <tr> | |
| <td class="num">13</td> | |
| <td class="num">13</td> | |
| <td class="fab-lb__sys-cell">stable-ts</td> | |
| <td>Open</td> | |
| <td class="num">0.236</td> | |
| <td class="num">0.228–0.244</td> | |
| <td class="num">0.015</td> | |
| <td class="num">0.193</td><td class="num">0.204</td><td class="num">0.278</td><td class="num">0.270</td> | |
| <td class="num">97.8</td><td class="num">70.8</td> | |
| <td class="num">–</td> | |
| </tr> | |
| <tr> | |
| <td class="num">14</td> | |
| <td class="num">14</td> | |
| <td class="fab-lb__sys-cell">NVIDIA NeMo-FA †</td> | |
| <td>Open</td> | |
| <td class="num">0.147</td> | |
| <td class="num">0.140–0.154</td> | |
| <td class="num">0.008</td> | |
| <td class="num">0.154</td><td class="num">0.154</td><td class="num">0.139</td><td class="num">0.140</td> | |
| <td class="num">79.3</td><td class="num">88.8</td> | |
| <td class="num">–</td> | |
| </tr> | |
| <tr class="is-withdrawn"> | |
| <td class="num">–</td> | |
| <td class="num">–</td> | |
| <td class="fab-lb__sys-cell">Olewave <a href="https://olewave.com/en/olign-olewaves-lancet-accurate-speech-to-text-forced-alignment-service/">Olign</a> 0.9</td> | |
| <td>Commercial</td> | |
| <td class="num">0.704</td> | |
| <td class="num">0.694–0.712</td> | |
| <td class="num">0.031</td> | |
| <td class="num">0.791</td><td class="num">0.697</td><td class="num">0.703</td><td class="num">0.623</td> | |
| <td class="num">18.3</td><td class="num">20.8</td> | |
| <td class="num">–</td> | |
| </tr> | |
| </tbody> | |
| </table> | |
| </div> | |
| <p class="fab-lb__drop">F1 Drop is how far an aligner's mean F1 falls when it is handed the words Qwen3-ASR recognized in place of the reference transcript. MFA 3.4, for example, scores 0.658 on the reference transcript and 0.628 on recognized words, an F1 Drop of 0.030.</p> | |
| <p class="fab-lb__notes">⚠ MAPS was trained on 7 of the 8 speakers in <a href="https://github.com/olewave/fa-bench/">FA-Bench</a>'s Buckeye test set, so most of its Buckeye test audio comes from speakers it was trained on.</p><p class="fab-lb__notes">⚠ CrisperWhisper's authors used TIMIT to build its word timing and don't say which TIMIT files. <a href="https://github.com/olewave/fa-bench/">FA-Bench</a> tests on TIMIT's test files, so if the authors used those too, its TIMIT scores come from audio it has already seen.</p><p class="fab-lb__notes">† NeMo-FA here is its 80 ms model. The 40 ms model's mean F1 is 0.189.</p><p class="fab-lb__notes">Qwen3-ASR's words have a WER of 2.0% on TIMIT core test and 10.2% on Buckeye test on clean audio. NeuFA was not run on recognized words. <a href="https://olewave.com/en/olign-olewaves-lancet-accurate-speech-to-text-forced-alignment-service/">Olign</a> v1 was not run on recognized words, so the unranked <a href="https://olewave.com/en/olign-olewaves-lancet-accurate-speech-to-text-forced-alignment-service/">Olign</a> row here is <a href="https://olewave.com/en/olign-olewaves-lancet-accurate-speech-to-text-forced-alignment-service/">Olign</a> 0.9.</p><p class="fab-lb__notes">ElevenLabs-FA is forced-alignment, API v1. Olewave <a href="https://olewave.com/en/olign-olewaves-lancet-accurate-speech-to-text-forced-alignment-service/">Olign</a> 0.9 is olign-0.9.</p><p class="fab-lb__notes"><a href="https://olewave.com/en/olign-olewaves-lancet-accurate-speech-to-text-forced-alignment-service/">Olign</a> no longer takes part in <a href="https://github.com/olewave/fa-bench/">FA-Bench</a>. Its last results, from October 2026, are listed unranked at the bottom of the table.</p> | |
| <p class="fab-lb__source"><a href="https://olewave.com/media/blog/fa-bench-leaderboard/track1_word_f1_recognized.csv?v=1793210021" download>Download the CSV</a>.</p> | |
| </section> | |
| <section class="fab-lb__static"> | |
| <h2>Forced alignment with recognized words, phone boundary F1 at 20 ms</h2> | |
| <div class="fab-lb__table-wrap"> | |
| <table class="fab-lb__tbl"> | |
| <thead><tr> | |
| <th class="num" scope="col">Rank</th><th class="num" scope="col">Range</th><th scope="col">System</th><th scope="col">Access</th> | |
| <th class="num" scope="col">Mean F1</th><th class="num" scope="col">95% interval</th><th class="num" scope="col">F1 Drop</th><th class="num" scope="col">TIMIT clean</th><th class="num" scope="col">TIMIT noisy</th> | |
| <th class="num" scope="col">Buckeye clean</th><th class="num" scope="col">Buckeye noisy</th> | |
| <th class="num" scope="col">PER TIMIT</th><th class="num" scope="col">PER Buckeye</th> | |
| <th class="num" scope="col">Price, $ per audio hour</th> | |
| </tr></thead> | |
| <tbody> | |
| <tr> | |
| <td class="num">1</td> | |
| <td class="num">1</td> | |
| <td class="fab-lb__sys-cell">MFA 3.4</td> | |
| <td>Open</td> | |
| <td class="num">0.428</td> | |
| <td class="num">0.423–0.434</td> | |
| <td class="num">0.008</td> | |
| <td class="num">0.394</td><td class="num">0.372</td><td class="num">0.502</td><td class="num">0.446</td> | |
| <td class="num">34.8</td><td class="num">28.2</td> | |
| <td class="num">–</td> | |
| </tr> | |
| <tr> | |
| <td class="num">2</td> | |
| <td class="num">2</td> | |
| <td class="fab-lb__sys-cell">PocketSphinx</td> | |
| <td>Open</td> | |
| <td class="num">0.395</td> | |
| <td class="num">0.389–0.401</td> | |
| <td class="num">0.007</td> | |
| <td class="num">0.337</td><td class="num">0.325</td><td class="num">0.482</td><td class="num">0.436</td> | |
| <td class="num">34.0</td><td class="num">27.2</td> | |
| <td class="num">–</td> | |
| </tr> | |
| <tr> | |
| <td class="num">3</td> | |
| <td class="num">3</td> | |
| <td class="fab-lb__sys-cell">MAPS ⚠</td> | |
| <td>Open</td> | |
| <td class="num">0.378</td> | |
| <td class="num">0.372–0.384</td> | |
| <td class="num">0.005</td> | |
| <td class="num">0.455</td><td class="num">0.319</td><td class="num">0.435</td><td class="num">0.303</td> | |
| <td class="num">30.7</td><td class="num">35.8</td> | |
| <td class="num">–</td> | |
| </tr> | |
| <tr> | |
| <td class="num">4</td> | |
| <td class="num">4</td> | |
| <td class="fab-lb__sys-cell">Charsiu</td> | |
| <td>Open</td> | |
| <td class="num">0.369</td> | |
| <td class="num">0.363–0.375</td> | |
| <td class="num">0.003</td> | |
| <td class="num">0.361</td><td class="num">0.347</td><td class="num">0.417</td><td class="num">0.352</td> | |
| <td class="num">30.3</td><td class="num">32.7</td> | |
| <td class="num">–</td> | |
| </tr> | |
| <tr> | |
| <td class="num">5</td> | |
| <td class="num">5</td> | |
| <td class="fab-lb__sys-cell">TorchAudio</td> | |
| <td>Open</td> | |
| <td class="num">0.230</td> | |
| <td class="num">0.226–0.235</td> | |
| <td class="num">0.005</td> | |
| <td class="num">0.207</td><td class="num">0.205</td><td class="num">0.260</td><td class="num">0.250</td> | |
| <td class="num">35.9</td><td class="num">30.0</td> | |
| <td class="num">–</td> | |
| </tr> | |
| <tr> | |
| <td class="num">6</td> | |
| <td class="num">6</td> | |
| <td class="fab-lb__sys-cell">BFA</td> | |
| <td>Open</td> | |
| <td class="num">0.213</td> | |
| <td class="num">0.208–0.217</td> | |
| <td class="num">0.004</td> | |
| <td class="num">0.183</td><td class="num">0.177</td><td class="num">0.253</td><td class="num">0.238</td> | |
| <td class="num">36.2</td><td class="num">33.6</td> | |
| <td class="num">–</td> | |
| </tr> | |
| <tr class="is-withdrawn"> | |
| <td class="num">–</td> | |
| <td class="num">–</td> | |
| <td class="fab-lb__sys-cell">Olewave <a href="https://olewave.com/en/olign-olewaves-lancet-accurate-speech-to-text-forced-alignment-service/">Olign</a> 0.9</td> | |
| <td>Commercial</td> | |
| <td class="num">0.447</td> | |
| <td class="num">0.440–0.453</td> | |
| <td class="num">0.005</td> | |
| <td class="num">0.410</td><td class="num">0.393</td><td class="num">0.513</td><td class="num">0.470</td> | |
| <td class="num">35.4</td><td class="num">29.2</td> | |
| <td class="num">–</td> | |
| </tr> | |
| </tbody> | |
| </table> | |
| </div> | |
| <p class="fab-lb__drop">F1 Drop is how far an aligner's mean F1 falls when it is handed the words Qwen3-ASR recognized in place of the reference transcript. MFA 3.4, for example, scores 0.436 on the reference transcript and 0.428 on recognized words, an F1 Drop of 0.008.</p> | |
| <p class="fab-lb__notes">⚠ MAPS was trained on 7 of the 8 speakers in <a href="https://github.com/olewave/fa-bench/">FA-Bench</a>'s Buckeye test set, so most of its Buckeye test audio comes from speakers it was trained on.</p><p class="fab-lb__notes">Qwen3-ASR's words have a WER of 2.0% on TIMIT core test and 10.2% on Buckeye test on clean audio. NeuFA was not run on recognized words. <a href="https://olewave.com/en/olign-olewaves-lancet-accurate-speech-to-text-forced-alignment-service/">Olign</a> v1 was not run on recognized words, so the unranked <a href="https://olewave.com/en/olign-olewaves-lancet-accurate-speech-to-text-forced-alignment-service/">Olign</a> row here is <a href="https://olewave.com/en/olign-olewaves-lancet-accurate-speech-to-text-forced-alignment-service/">Olign</a> 0.9.</p><p class="fab-lb__notes">Olewave <a href="https://olewave.com/en/olign-olewaves-lancet-accurate-speech-to-text-forced-alignment-service/">Olign</a> 0.9 is olign-0.9.</p><p class="fab-lb__notes"><a href="https://olewave.com/en/olign-olewaves-lancet-accurate-speech-to-text-forced-alignment-service/">Olign</a> no longer takes part in <a href="https://github.com/olewave/fa-bench/">FA-Bench</a>. Its last results, from October 2026, are listed unranked at the bottom of the table.</p> | |
| <p class="fab-lb__source"><a href="https://olewave.com/media/blog/fa-bench-leaderboard/track1_phone_f1_recognized.csv?v=1793210021" download>Download the CSV</a>.</p> | |
| </section> | |
| </div> | |
| <script id="fab-i18n" type="application/json">{}</script> | |
| <script src="leaderboard.js" defer></script> | |
| <aside class="blog-article__cta"> | |
| <p><strong><a href="https://olewave.com/en/olign-olewaves-lancet-accurate-speech-to-text-forced-alignment-service/">Olign</a> v1 is available as an API.</strong> Send a recording with or without its transcript, and every word and phone comes back with its start and end time and a score from 0 to 100, in US and UK English. Anyone can sign up at <a href="https://tycho.olewave.com">tycho.olewave.com</a> and get 300 free credits, or 3,000 with an invitation code, which you can <a href="https://olewave.com/en/contact_us/">request from us</a>.</p> | |
| </aside> | |
| </div> | |
| </div> | |
| </div> | |
| </article> | |
| </div> | |
| </body> | |
| </html> | |