Hugging Face's logo

vertox-ai
/
speech-text-data-aligners

NeMo
English
audio
speech
forced-alignment
reference-text-alignment
speech-processing
mfa
whisperx
python
metric:mae
Eval Results (legacy)
Model card Files
xet
 Community
speech-text-data-aligners / docs /EVALUATION.md
pltobing's picture
pltobing
docs: refine aligner documentation
a3b146e
preview code
|
Raw History Blame Contribute Delete
10.3 kB

Evaluation

This page is the canonical comparison of the alignment models and configurations in this repository. It distinguishes manual-gold accuracy, structural success, pairwise disagreement, diagnostic behavior, runtime observations, and offline readiness agreement. Those evidence types are not interchangeable.

Evaluation Scope

Evidence set Rows Available reference Purpose
Five-row manual gold 5 first_word_text, first_word_end_s Limited timing accuracy
Full metadata set 36 Authoritative transcripts and WAV bounds Structural success and cross-backend disagreement
W1-T semantic-readiness timing 36 Frozen semantic boundary tokens Offline automatic END estimates and agreement

The metadata evaluation aligns the supplied en_transcript; it does not score ASR transcription. The five-row file contains no gold for first_word_start_s, last_word_start_s, or last_word_end_s.

Compared Models and Configurations

Backend Runtime Model identity Dictionary/model revision Device and cache
MFA MFA 3.4.2; Kaldi 5.5.1172; Kalpy 0.10.5; OpenFst 1.8.4 MontrealCorpusTools/english_mfa model 3.3.0 Revision 1009e3c6c689f10875146a5e4e5c2233f8eaf922; english_us_mfa; G2P enabled CPU; cold/model-cache conditions vary by run
WhisperX WhisperX 3.8.6; Torch/Torchaudio 2.8.0 WAV2VEC2_ASR_BASE_960H Torchaudio 2.8.0 bundle; observed asset SHA-256 488fd4f...c945d CPU; prepopulated hash-verified cache in accepted runs
NeMo / NFA NeMo 3.0.0 source fd6a877539710e2b98f28c43272ff81312f83417 stt_en_fastconformer_hybrid_large_pc Local checkpoint hash 37afe662...2c8b61; no registry revision recorded NVIDIA A10G in accepted empirical runs

The full unabbreviated identities and hashes are in config.json, the run receipts, and Configuration.

Metric Definitions

Metric Definition Missing/failure behavior
Success A backend returned a complete result that passed lexical/timestamp validation Failure is counted and never receives invented timing
Exact first-word text Output first_word_text exactly equals the reference surface Failed rows cannot match
First-word-end absolute error abs(predicted_first_word_end_s - reference_first_word_end_s) Empty reference is counted as missing and excluded
MAE Mean of available first-word-end absolute errors No scorable reference yields null
Median / maximum Median and maximum available first-word-end absolute error Same exclusion as MAE
Pairwise boundary difference Absolute difference between two successful backend boundaries Only rows successful in both backends
Material disagreement Any first/last boundary differs by at least the declared threshold Diagnostic review queue, not an error rate
Runtime Measured service/batch execution time Observed only; not normalized across hardware/cache

Benchmark reports expose success/failure counts, missing-reference counts, exact-text rate, first-end MAE/median/maximum, runtime, and per-boundary pairwise MAE. Historical comparison artifacts additionally calculate pairwise median, p95, maximum, and threshold counts.

Five-Row Manual Gold

Evidence date: 2026-09-06 accepted comparison. Sample size: 5.

Backend Configuration Success Exact first text First-end MAE Median Maximum
MFA english_mfa@1009e3c..., english_us_mfa 5/5 5/5 0.0162 s 0.0110 s 0.0300 s
WhisperX WAV2VEC2_ASR_BASE_960H, CPU 5/5 5/5 0.0154 s 0.0140 s 0.0310 s
NeMo stt_en_fastconformer_hybrid_large_pc, A10G 5/5 5/5 0.2210 s 0.0100 s 0.9200 s

Interpretation:

  • MFA and WhisperX are effectively tied on the only timing gold.
  • WhisperX MAE is lower by 0.0008 s; the sample is too small to rank by this alone.
  • MFA has slightly lower median and maximum error.
  • NeMo has low median error but a 0.920 s maximum on EN0949, producing its larger MAE.
  • These results do not establish accuracy for first-word start or either last boundary.

Full-36 Structural Comparison

All three backends produced 36/36 candidates. Every accepted candidate preserved source cells/order, the exact six-column extension, authoritative first/last lexical surfaces, and finite WAV-bounded monotonic timestamps.

Pairwise Absolute Differences

Pair Boundary Mean Median p95 Maximum
MFA / WhisperX first start 0.059306 s 0.051000 s 0.116750 s 0.132000 s
MFA / WhisperX first end 0.018694 s 0.015000 s 0.039000 s 0.078000 s
MFA / WhisperX last start 0.211611 s 0.064000 s 1.062500 s 1.545000 s
MFA / WhisperX last end 0.325083 s 0.128000 s 1.449750 s 1.835000 s
MFA / NeMo first start 0.665556 s 0.325000 s 1.430000 s 2.120000 s
MFA / NeMo first end 0.292500 s 0.170000 s 1.250000 s 1.510000 s
MFA / NeMo last start 0.137778 s 0.160000 s 0.225000 s 0.250000 s
MFA / NeMo last end 0.104444 s 0.100000 s 0.222500 s 0.260000 s
WhisperX / NeMo first start 0.724861 s 0.371500 s 1.453000 s 2.122000 s
WhisperX / NeMo first end 0.292639 s 0.151000 s 1.252250 s 1.503000 s
WhisperX / NeMo last start 0.259500 s 0.106000 s 1.247500 s 1.795000 s
WhisperX / NeMo last end 0.398861 s 0.202000 s 1.534750 s 1.955000 s

p95 uses linear interpolation at rank (n - 1) * 0.95.

Material-Disagreement Counts

Counts are rows where at least one of the four boundaries reaches the threshold:

Pair >=0.250 s >=0.500 s >=1.000 s
MFA / WhisperX 7 6 5
MFA / NeMo 28 17 12
WhisperX / NeMo 32 21 16

MFA and WhisperX are close at first-word boundaries. All seven of their material rows are driven by last-word boundaries. These differences identify review cases; they do not show which backend is acoustically correct.

Diagnostic Boundary Signals

Signal MFA WhisperX NeMo
Exact zero first-word starts 0/36 0/36 17/36
First-word duration p95 0.457 s 0.400 s 1.560 s
First-word duration maximum 0.720 s 0.621 s 2.160 s
Final word within 50 ms of WAV end 0/36 0/36 0/36

NeMo's repeated zero starts and the EN0949 first-end outlier are material operational warnings. MFA and NeMo have smaller last-boundary disagreement from each other than either has with WhisperX on the largest cases. This is still concordance evidence, not manual ground truth.

Runtime Observations

Backend Observed run Runtime Conditions
MFA Accepted original five-row cold run 274.9569 s Cold run; later run observed 79.5288 s under different cache/host conditions
WhisperX Accepted five-row run 28.5392 s CPU, prepopulated hash-verified model cache
NeMo Accepted five-row run 13.7050 s NVIDIA A10G after smoke; caches may have been warm

Runtime comparisons are secondary. Host, device, process overhead, model download, and cache state were not controlled. Do not convert these observations into a general speed ranking.

W1-T Semantic-Readiness Evidence

W1-T maps frozen English semantic-boundary tokens to automatic acoustic END estimates. It does not measure human Russian onset and does not adjudicate semantic boundaries.

Metric Observed result
Cases 36
Mapping 36 EXACT
Confidence 1 HIGH, 35 MEDIUM
LOW / SINGLE_BACKEND / UNRESOLVED 0 / 0 / 0
Cross-backend range p50 22.5 ms
Cross-backend range p95 86.5 ms
Cross-backend maximum range 127 ms
Semantic-ready END p50 2.1765 s
Semantic-ready END p95 3.316625 s

Consensus is the median of valid independent backend END estimates. HIGH requires exactly three valid backends and range <=100 ms; MEDIUM requires at least two and range <=250 ms; LOW requires at least two and range >250 ms. Agreement is not timing accuracy. See W1-T Semantic Readiness for reuse, provenance, mapping, and materialization rules.

Operational Selection

MFA is the selected operational backend as of 2026-09-06. The decision combines:

  1. strict 36/36 eligibility;
  2. a first-word-end MAE effectively tied with WhisperX;
  3. no repeated zero first-word starts;
  4. diagnostic last-boundary support on the largest disputed cases;
  5. reproducible pinned runtime/model identity.

This is an operational choice from limited evidence, not a claim of universal timing superiority. WhisperX remains the accepted alternative. NeMo remains usable behind the same protocol where its pinned GPU runtime/model are available.

Canonical Artifacts

Artifact Purpose
results/final/final-annotated-36.csv Canonical MFA annotation, 36/36
results/final/final-annotated-36.receipt.json Import/final hashes and evidence links
results/final/final-aligner-selection.{json,md} Canonical selection record
results/postfinal/three-backend-full-36/ Full comparison, pairwise tables, runtime and systematic analysis
results/postfinal/mfa-full-36/ MFA full-run validation and diagnostics
results/stage5/ Historical WhisperX full-run and original acceptance evidence
results/w1t/ Offline semantic-readiness sidecars and receipts

The canonical CSV SHA-256 is 37bcb85fc22fb142dae40453b90dd3b427bbfd657861bee7529cd433915535a8. Repository CSV copies use LF line endings; import receipts preserve the reviewed external artifact hash where line endings differed.

Limitations

  • Manual gold covers only first-word text/end on five rows.
  • First-word start and last-word boundaries have no manual ground truth.
  • Pairwise agreement cannot establish acoustic accuracy.
  • The 0.0008 s MFA/WhisperX MAE difference is not decision-grade.
  • Full-corpus pairwise analysis is diagnostic and threshold-dependent.
  • Runtime observations are not hardware/cache controlled.
  • Accents, broader acoustic conditions, repeated runs, and larger human-annotated sets remain outside the current evidence boundary.