| # Evaluation |
|
|
| This page is the canonical comparison of the alignment models and configurations in |
| this repository. It distinguishes manual-gold accuracy, structural success, pairwise |
| disagreement, diagnostic behavior, runtime observations, and offline readiness |
| agreement. Those evidence types are not interchangeable. |
|
|
| ## Evaluation Scope |
|
|
| | Evidence set | Rows | Available reference | Purpose | |
| |---|---:|---|---| |
| | Five-row manual gold | 5 | `first_word_text`, `first_word_end_s` | Limited timing accuracy | |
| | Full metadata set | 36 | Authoritative transcripts and WAV bounds | Structural success and cross-backend disagreement | |
| | W1-T semantic-readiness timing | 36 | Frozen semantic boundary tokens | Offline automatic END estimates and agreement | |
|
|
| The metadata evaluation aligns the supplied `en_transcript`; it does not score ASR |
| transcription. The five-row file contains no gold for `first_word_start_s`, |
| `last_word_start_s`, or `last_word_end_s`. |
|
|
| ## Compared Models and Configurations |
|
|
| | Backend | Runtime | Model identity | Dictionary/model revision | Device and cache | |
| |---|---|---|---|---| |
| | MFA | MFA 3.4.2; Kaldi 5.5.1172; Kalpy 0.10.5; OpenFst 1.8.4 | `MontrealCorpusTools/english_mfa` model 3.3.0 | Revision `1009e3c6c689f10875146a5e4e5c2233f8eaf922`; `english_us_mfa`; G2P enabled | CPU; cold/model-cache conditions vary by run | |
| | WhisperX | WhisperX 3.8.6; Torch/Torchaudio 2.8.0 | `WAV2VEC2_ASR_BASE_960H` | Torchaudio 2.8.0 bundle; observed asset SHA-256 `488fd4f...c945d` | CPU; prepopulated hash-verified cache in accepted runs | |
| | NeMo / NFA | NeMo 3.0.0 source `fd6a877539710e2b98f28c43272ff81312f83417` | `stt_en_fastconformer_hybrid_large_pc` | Local checkpoint hash `37afe662...2c8b61`; no registry revision recorded | NVIDIA A10G in accepted empirical runs | |
|
|
| The full unabbreviated identities and hashes are in [`config.json`](../config.json), |
| the run receipts, and [Configuration](CONFIGURATION.md). |
|
|
| ## Metric Definitions |
|
|
| | Metric | Definition | Missing/failure behavior | |
| |---|---|---| |
| | Success | A backend returned a complete result that passed lexical/timestamp validation | Failure is counted and never receives invented timing | |
| | Exact first-word text | Output `first_word_text` exactly equals the reference surface | Failed rows cannot match | |
| | First-word-end absolute error | `abs(predicted_first_word_end_s - reference_first_word_end_s)` | Empty reference is counted as missing and excluded | |
| | MAE | Mean of available first-word-end absolute errors | No scorable reference yields null | |
| | Median / maximum | Median and maximum available first-word-end absolute error | Same exclusion as MAE | |
| | Pairwise boundary difference | Absolute difference between two successful backend boundaries | Only rows successful in both backends | |
| | Material disagreement | Any first/last boundary differs by at least the declared threshold | Diagnostic review queue, not an error rate | |
| | Runtime | Measured service/batch execution time | Observed only; not normalized across hardware/cache | |
|
|
| Benchmark reports expose success/failure counts, missing-reference counts, exact-text |
| rate, first-end MAE/median/maximum, runtime, and per-boundary pairwise MAE. Historical |
| comparison artifacts additionally calculate pairwise median, p95, maximum, and |
| threshold counts. |
|
|
| ## Five-Row Manual Gold |
|
|
| Evidence date: 2026-09-06 accepted comparison. Sample size: 5. |
|
|
| | Backend | Configuration | Success | Exact first text | First-end MAE | Median | Maximum | |
| |---|---|---:|---:|---:|---:|---:| |
| | MFA | `english_mfa@1009e3c...`, `english_us_mfa` | 5/5 | 5/5 | **0.0162 s** | **0.0110 s** | **0.0300 s** | |
| | WhisperX | `WAV2VEC2_ASR_BASE_960H`, CPU | 5/5 | 5/5 | **0.0154 s** | **0.0140 s** | **0.0310 s** | |
| | NeMo | `stt_en_fastconformer_hybrid_large_pc`, A10G | 5/5 | 5/5 | **0.2210 s** | **0.0100 s** | **0.9200 s** | |
|
|
| Interpretation: |
|
|
| - MFA and WhisperX are effectively tied on the only timing gold. |
| - WhisperX MAE is lower by 0.0008 s; the sample is too small to rank by this alone. |
| - MFA has slightly lower median and maximum error. |
| - NeMo has low median error but a 0.920 s maximum on `EN0949`, producing its larger MAE. |
| - These results do not establish accuracy for first-word start or either last boundary. |
|
|
| ## Full-36 Structural Comparison |
|
|
| All three backends produced 36/36 candidates. Every accepted candidate preserved |
| source cells/order, the exact six-column extension, authoritative first/last lexical |
| surfaces, and finite WAV-bounded monotonic timestamps. |
|
|
| ### Pairwise Absolute Differences |
|
|
| | Pair | Boundary | Mean | Median | p95 | Maximum | |
| |---|---|---:|---:|---:|---:| |
| | MFA / WhisperX | first start | 0.059306 s | 0.051000 s | 0.116750 s | 0.132000 s | |
| | MFA / WhisperX | first end | 0.018694 s | 0.015000 s | 0.039000 s | 0.078000 s | |
| | MFA / WhisperX | last start | 0.211611 s | 0.064000 s | 1.062500 s | 1.545000 s | |
| | MFA / WhisperX | last end | 0.325083 s | 0.128000 s | 1.449750 s | 1.835000 s | |
| | MFA / NeMo | first start | 0.665556 s | 0.325000 s | 1.430000 s | 2.120000 s | |
| | MFA / NeMo | first end | 0.292500 s | 0.170000 s | 1.250000 s | 1.510000 s | |
| | MFA / NeMo | last start | 0.137778 s | 0.160000 s | 0.225000 s | 0.250000 s | |
| | MFA / NeMo | last end | 0.104444 s | 0.100000 s | 0.222500 s | 0.260000 s | |
| | WhisperX / NeMo | first start | 0.724861 s | 0.371500 s | 1.453000 s | 2.122000 s | |
| | WhisperX / NeMo | first end | 0.292639 s | 0.151000 s | 1.252250 s | 1.503000 s | |
| | WhisperX / NeMo | last start | 0.259500 s | 0.106000 s | 1.247500 s | 1.795000 s | |
| | WhisperX / NeMo | last end | 0.398861 s | 0.202000 s | 1.534750 s | 1.955000 s | |
|
|
| p95 uses linear interpolation at rank `(n - 1) * 0.95`. |
|
|
| ### Material-Disagreement Counts |
|
|
| Counts are rows where at least one of the four boundaries reaches the threshold: |
|
|
| | Pair | >=0.250 s | >=0.500 s | >=1.000 s | |
| |---|---:|---:|---:| |
| | MFA / WhisperX | 7 | 6 | 5 | |
| | MFA / NeMo | 28 | 17 | 12 | |
| | WhisperX / NeMo | 32 | 21 | 16 | |
|
|
| MFA and WhisperX are close at first-word boundaries. All seven of their material rows |
| are driven by last-word boundaries. These differences identify review cases; they do |
| not show which backend is acoustically correct. |
|
|
| ## Diagnostic Boundary Signals |
|
|
| | Signal | MFA | WhisperX | NeMo | |
| |---|---:|---:|---:| |
| | Exact zero first-word starts | 0/36 | 0/36 | 17/36 | |
| | First-word duration p95 | 0.457 s | 0.400 s | 1.560 s | |
| | First-word duration maximum | 0.720 s | 0.621 s | 2.160 s | |
| | Final word within 50 ms of WAV end | 0/36 | 0/36 | 0/36 | |
|
|
| NeMo's repeated zero starts and the `EN0949` first-end outlier are material operational |
| warnings. MFA and NeMo have smaller last-boundary disagreement from each other than |
| either has with WhisperX on the largest cases. This is still concordance evidence, not |
| manual ground truth. |
|
|
| ## Runtime Observations |
|
|
| | Backend | Observed run | Runtime | Conditions | |
| |---|---|---:|---| |
| | MFA | Accepted original five-row cold run | 274.9569 s | Cold run; later run observed 79.5288 s under different cache/host conditions | |
| | WhisperX | Accepted five-row run | 28.5392 s | CPU, prepopulated hash-verified model cache | |
| | NeMo | Accepted five-row run | 13.7050 s | NVIDIA A10G after smoke; caches may have been warm | |
|
|
| Runtime comparisons are secondary. Host, device, process overhead, model download, and |
| cache state were not controlled. Do not convert these observations into a general |
| speed ranking. |
|
|
| ## W1-T Semantic-Readiness Evidence |
|
|
| W1-T maps frozen English semantic-boundary tokens to automatic acoustic END estimates. |
| It does not measure human Russian onset and does not adjudicate semantic boundaries. |
|
|
| | Metric | Observed result | |
| |---|---| |
| | Cases | 36 | |
| | Mapping | 36 EXACT | |
| | Confidence | 1 HIGH, 35 MEDIUM | |
| | LOW / SINGLE_BACKEND / UNRESOLVED | 0 / 0 / 0 | |
| | Cross-backend range p50 | 22.5 ms | |
| | Cross-backend range p95 | 86.5 ms | |
| | Cross-backend maximum range | 127 ms | |
| | Semantic-ready END p50 | 2.1765 s | |
| | Semantic-ready END p95 | 3.316625 s | |
| |
| Consensus is the median of valid independent backend END estimates. HIGH requires |
| exactly three valid backends and range <=100 ms; MEDIUM requires at least two and range |
| <=250 ms; LOW requires at least two and range >250 ms. Agreement is not timing |
| accuracy. See [W1-T Semantic Readiness](W1_T_SEMANTIC_READINESS.md) for reuse, |
| provenance, mapping, and materialization rules. |
| |
| ## Operational Selection |
| |
| MFA is the selected operational backend as of 2026-09-06. The decision combines: |
| |
| 1. strict 36/36 eligibility; |
| 2. a first-word-end MAE effectively tied with WhisperX; |
| 3. no repeated zero first-word starts; |
| 4. diagnostic last-boundary support on the largest disputed cases; |
| 5. reproducible pinned runtime/model identity. |
| |
| This is an operational choice from limited evidence, not a claim of universal timing |
| superiority. WhisperX remains the accepted alternative. NeMo remains usable behind the |
| same protocol where its pinned GPU runtime/model are available. |
| |
| ## Canonical Artifacts |
| |
| | Artifact | Purpose | |
| |---|---| |
| | `results/final/final-annotated-36.csv` | Canonical MFA annotation, 36/36 | |
| | `results/final/final-annotated-36.receipt.json` | Import/final hashes and evidence links | |
| | `results/final/final-aligner-selection.{json,md}` | Canonical selection record | |
| | `results/postfinal/three-backend-full-36/` | Full comparison, pairwise tables, runtime and systematic analysis | |
| | `results/postfinal/mfa-full-36/` | MFA full-run validation and diagnostics | |
| | `results/stage5/` | Historical WhisperX full-run and original acceptance evidence | |
| | `results/w1t/` | Offline semantic-readiness sidecars and receipts | |
| |
| The canonical CSV SHA-256 is |
| `37bcb85fc22fb142dae40453b90dd3b427bbfd657861bee7529cd433915535a8`. |
| Repository CSV copies use LF line endings; import receipts preserve the reviewed |
| external artifact hash where line endings differed. |
| |
| ## Limitations |
| |
| - Manual gold covers only first-word text/end on five rows. |
| - First-word start and last-word boundaries have no manual ground truth. |
| - Pairwise agreement cannot establish acoustic accuracy. |
| - The 0.0008 s MFA/WhisperX MAE difference is not decision-grade. |
| - Full-corpus pairwise analysis is diagnostic and threshold-dependent. |
| - Runtime observations are not hardware/cache controlled. |
| - Accents, broader acoustic conditions, repeated runs, and larger human-annotated sets |
| remain outside the current evidence boundary. |
| |
