speech-text-data-aligners / docs /FINAL_ALIGNER_SELECTION.md
pltobing's picture
docs(aligners): finalize three-backend selection
09db84f
|
Raw History Blame Contribute Delete
1.87 kB

Final Aligner Selection

Decision: SELECTED_MFA (2026-09-06).

This decision supersedes the 2026-09-04 SELECTED_WHISPERX operational choice after the previously missing strict MFA full-36 evaluation was completed. Historical Stage 5 evidence remains unchanged.

MFA, WhisperX, and NeMo now all have successful 36/36 candidates. On the five rows with manual first-word gold, MFA and WhisperX remain effectively tied: MFA first-word-end MAE is 0.0162 s (median 0.0110, max 0.0300), while WhisperX is 0.0154 s (median 0.0140, max 0.0310). NeMo is 0.2210 s with a 0.9200 s maximum error on EN0949.

Across all 36 rows, MFA and WhisperX differ by at least 0.250 s on 7 rows, versus 28 for MFA/NeMo and 32 for WhisperX/NeMo. MFA and WhisperX are especially close at first-word boundaries (mean absolute differences 0.059306 s start and 0.018694 s end). Their material disagreements are concentrated at last-word boundaries. On the largest disputed cases, retained MFA forced-alignment evidence and independent NeMo last-boundary timings support the later MFA placements. This is diagnostic evidence rather than manual ground truth.

The canonical final output is results/final/final-annotated-36.csv with SHA-256 37bcb85fc22fb142dae40453b90dd3b427bbfd657861bee7529cd433915535a8 and is byte-identical to results/postfinal/mfa-full-36/annotated-36-mfa.csv.

Limitations

  • Manual gold covers only first_word_text and first_word_end_s on five rows.
  • First-word start and both last-word boundaries have no manual ground truth.
  • Pairwise agreement is not accuracy.
  • The 0.0008 s MFA/WhisperX five-row MAE difference is too small to rank the systems by itself.
  • Runtime results are from different hosts/cache states and are secondary evidence.
  • This selection is an operational choice from the available evidence, not a claim of universal timing superiority.