Blame for docs/EVALUATION.md

# Evaluation

This page is the canonical comparison of the alignment models and configurations in
this repository. It distinguishes manual-gold accuracy, structural success, pairwise
disagreement, diagnostic behavior, runtime observations, and offline readiness
agreement. Those evidence types are not interchangeable.

## Evaluation Scope

| Evidence set | Rows | Available reference | Purpose |
|---|---:|---|---|
| Five-row manual gold | 5 | `first_word_text`, `first_word_end_s` | Limited timing accuracy |
| Full metadata set | 36 | Authoritative transcripts and WAV bounds | Structural success and cross-backend disagreement |
| W1-T semantic-readiness timing | 36 | Frozen semantic boundary tokens | Offline automatic END estimates and agreement |

The metadata evaluation aligns the supplied `en_transcript`; it does not score ASR
transcription. The five-row file contains no gold for `first_word_start_s`,
`last_word_start_s`, or `last_word_end_s`.

## Compared Models and Configurations

| Backend | Runtime | Model identity | Dictionary/model revision | Device and cache |
|---|---|---|---|---|
| MFA | MFA 3.4.2; Kaldi 5.5.1172; Kalpy 0.10.5; OpenFst 1.8.4 | `MontrealCorpusTools/english_mfa` model 3.3.0 | Revision `1009e3c6c689f10875146a5e4e5c2233f8eaf922`; `english_us_mfa`; G2P enabled | CPU; cold/model-cache conditions vary by run |
| WhisperX | WhisperX 3.8.6; Torch/Torchaudio 2.8.0 | `WAV2VEC2_ASR_BASE_960H` | Torchaudio 2.8.0 bundle; observed asset SHA-256 `488fd4f...c945d` | CPU; prepopulated hash-verified cache in accepted runs |
| NeMo / NFA | NeMo 3.0.0 source `fd6a877539710e2b98f28c43272ff81312f83417` | `stt_en_fastconformer_hybrid_large_pc` | Local checkpoint hash `37afe662...2c8b61`; no registry revision recorded | NVIDIA A10G in accepted empirical runs |

The full unabbreviated identities and hashes are in [`config.json`](../config.json),
the run receipts, and [Configuration](CONFIGURATION.md).

## Metric Definitions

| Metric | Definition | Missing/failure behavior |
|---|---|---|
| Success | A backend returned a complete result that passed lexical/timestamp validation | Failure is counted and never receives invented timing |
| Exact first-word text | Output `first_word_text` exactly equals the reference surface | Failed rows cannot match |
| First-word-end absolute error | `abs(predicted_first_word_end_s - reference_first_word_end_s)` | Empty reference is counted as missing and excluded |
| MAE | Mean of available first-word-end absolute errors | No scorable reference yields null |
| Median / maximum | Median and maximum available first-word-end absolute error | Same exclusion as MAE |
| Pairwise boundary difference | Absolute difference between two successful backend boundaries | Only rows successful in both backends |
| Material disagreement | Any first/last boundary differs by at least the declared threshold | Diagnostic review queue, not an error rate |
| Runtime | Measured service/batch execution time | Observed only; not normalized across hardware/cache |

Benchmark reports expose success/failure counts, missing-reference counts, exact-text
rate, first-end MAE/median/maximum, runtime, and per-boundary pairwise MAE. Historical
comparison artifacts additionally calculate pairwise median, p95, maximum, and
threshold counts.

## Five-Row Manual Gold

Evidence date: 2026-09-06 accepted comparison. Sample size: 5.

| Backend | Configuration | Success | Exact first text | First-end MAE | Median | Maximum |
|---|---|---:|---:|---:|---:|---:|
| MFA | `english_mfa@1009e3c...`, `english_us_mfa` | 5/5 | 5/5 | **0.0162 s** | **0.0110 s** | **0.0300 s** |
| WhisperX | `WAV2VEC2_ASR_BASE_960H`, CPU | 5/5 | 5/5 | **0.0154 s** | **0.0140 s** | **0.0310 s** |
| NeMo | `stt_en_fastconformer_hybrid_large_pc`, A10G | 5/5 | 5/5 | **0.2210 s** | **0.0100 s** | **0.9200 s** |

Interpretation:

- MFA and WhisperX are effectively tied on the only timing gold.
- WhisperX MAE is lower by 0.0008 s; the sample is too small to rank by this alone.
- MFA has slightly lower median and maximum error.
- NeMo has low median error but a 0.920 s maximum on `EN0949`, producing its larger MAE.
- These results do not establish accuracy for first-word start or either last boundary.

## Full-36 Structural Comparison

All three backends produced 36/36 candidates. Every accepted candidate preserved
source cells/order, the exact six-column extension, authoritative first/last lexical
surfaces, and finite WAV-bounded monotonic timestamps.

### Pairwise Absolute Differences

| Pair | Boundary | Mean | Median | p95 | Maximum |
|---|---|---:|---:|---:|---:|
| MFA / WhisperX | first start | 0.059306 s | 0.051000 s | 0.116750 s | 0.132000 s |
| MFA / WhisperX | first end | 0.018694 s | 0.015000 s | 0.039000 s | 0.078000 s |
| MFA / WhisperX | last start | 0.211611 s | 0.064000 s | 1.062500 s | 1.545000 s |
| MFA / WhisperX | last end | 0.325083 s | 0.128000 s | 1.449750 s | 1.835000 s |
| MFA / NeMo | first start | 0.665556 s | 0.325000 s | 1.430000 s | 2.120000 s |
| MFA / NeMo | first end | 0.292500 s | 0.170000 s | 1.250000 s | 1.510000 s |
| MFA / NeMo | last start | 0.137778 s | 0.160000 s | 0.225000 s | 0.250000 s |
| MFA / NeMo | last end | 0.104444 s | 0.100000 s | 0.222500 s | 0.260000 s |
| WhisperX / NeMo | first start | 0.724861 s | 0.371500 s | 1.453000 s | 2.122000 s |
| WhisperX / NeMo | first end | 0.292639 s | 0.151000 s | 1.252250 s | 1.503000 s |
| WhisperX / NeMo | last start | 0.259500 s | 0.106000 s | 1.247500 s | 1.795000 s |
| WhisperX / NeMo | last end | 0.398861 s | 0.202000 s | 1.534750 s | 1.955000 s |

p95 uses linear interpolation at rank `(n - 1) * 0.95`.

### Material-Disagreement Counts

Counts are rows where at least one of the four boundaries reaches the threshold:

| Pair | >=0.250 s | >=0.500 s | >=1.000 s |
|---|---:|---:|---:|
| MFA / WhisperX | 7 | 6 | 5 |
| MFA / NeMo | 28 | 17 | 12 |
| WhisperX / NeMo | 32 | 21 | 16 |

MFA and WhisperX are close at first-word boundaries. All seven of their material rows
are driven by last-word boundaries. These differences identify review cases; they do
not show which backend is acoustically correct.

## Diagnostic Boundary Signals

| Signal | MFA | WhisperX | NeMo |
|---|---:|---:|---:|
| Exact zero first-word starts | 0/36 | 0/36 | 17/36 |
| First-word duration p95 | 0.457 s | 0.400 s | 1.560 s |
| First-word duration maximum | 0.720 s | 0.621 s | 2.160 s |
| Final word within 50 ms of WAV end | 0/36 | 0/36 | 0/36 |

NeMo's repeated zero starts and the `EN0949` first-end outlier are material operational
warnings. MFA and NeMo have smaller last-boundary disagreement from each other than
either has with WhisperX on the largest cases. This is still concordance evidence, not
manual ground truth.

## Runtime Observations

| Backend | Observed run | Runtime | Conditions |
|---|---|---:|---|
| MFA | Accepted original five-row cold run | 274.9569 s | Cold run; later run observed 79.5288 s under different cache/host conditions |
| WhisperX | Accepted five-row run | 28.5392 s | CPU, prepopulated hash-verified model cache |
| NeMo | Accepted five-row run | 13.7050 s | NVIDIA A10G after smoke; caches may have been warm |

Runtime comparisons are secondary. Host, device, process overhead, model download, and
cache state were not controlled. Do not convert these observations into a general
speed ranking.

## W1-T Semantic-Readiness Evidence

W1-T maps frozen English semantic-boundary tokens to automatic acoustic END estimates.
It does not measure human Russian onset and does not adjudicate semantic boundaries.

| Metric | Observed result |
|---|---|
| Cases | 36 |
| Mapping | 36 EXACT |
| Confidence | 1 HIGH, 35 MEDIUM |
| LOW / SINGLE_BACKEND / UNRESOLVED | 0 / 0 / 0 |
| Cross-backend range p50 | 22.5 ms |
| Cross-backend range p95 | 86.5 ms |
| Cross-backend maximum range | 127 ms |
| Semantic-ready END p50 | 2.1765 s |
| Semantic-ready END p95 | 3.316625 s |

Consensus is the median of valid independent backend END estimates. HIGH requires
exactly three valid backends and range <=100 ms; MEDIUM requires at least two and range
<=250 ms; LOW requires at least two and range >250 ms. Agreement is not timing
accuracy. See [W1-T Semantic Readiness](W1_T_SEMANTIC_READINESS.md) for reuse,
provenance, mapping, and materialization rules.

## Operational Selection

MFA is the selected operational backend as of 2026-09-06. The decision combines:

1. strict 36/36 eligibility;
2. a first-word-end MAE effectively tied with WhisperX;
3. no repeated zero first-word starts;
4. diagnostic last-boundary support on the largest disputed cases;
5. reproducible pinned runtime/model identity.

This is an operational choice from limited evidence, not a claim of universal timing
superiority. WhisperX remains the accepted alternative. NeMo remains usable behind the
same protocol where its pinned GPU runtime/model are available.

## Canonical Artifacts

| Artifact | Purpose |
|---|---|
| `results/final/final-annotated-36.csv` | Canonical MFA annotation, 36/36 |
| `results/final/final-annotated-36.receipt.json` | Import/final hashes and evidence links |
| `results/final/final-aligner-selection.{json,md}` | Canonical selection record |
| `results/postfinal/three-backend-full-36/` | Full comparison, pairwise tables, runtime and systematic analysis |
| `results/postfinal/mfa-full-36/` | MFA full-run validation and diagnostics |
| `results/stage5/` | Historical WhisperX full-run and original acceptance evidence |
| `results/w1t/` | Offline semantic-readiness sidecars and receipts |

The canonical CSV SHA-256 is
`37bcb85fc22fb142dae40453b90dd3b427bbfd657861bee7529cd433915535a8`.
Repository CSV copies use LF line endings; import receipts preserve the reviewed
external artifact hash where line endings differed.

## Limitations

- Manual gold covers only first-word text/end on five rows.
- First-word start and last-word boundaries have no manual ground truth.
- Pairwise agreement cannot establish acoustic accuracy.
- The 0.0008 s MFA/WhisperX MAE difference is not decision-grade.
- Full-corpus pairwise analysis is diagnostic and threshold-dependent.
- Runtime observations are not hardware/cache controlled.
- Accents, broader acoustic conditions, repeated runs, and larger human-annotated sets
  remain outside the current evidence boundary.