vertox-ai/speech-text-data-aligners

Speech-Text Data Aligners

Speech-Text Data Aligners (STA) is a library-first framework for forced alignment of WAV audio against a known, authoritative English transcript. It estimates when the supplied words occur in audio. It is not ASR and never replaces en_transcript with a recognized transcript.

The repository provides one backend-neutral Python contract for Montreal Forced Aligner (MFA), NVIDIA NeMo Forced Aligner (NFA), and WhisperX, plus deterministic annotation, benchmarking, and offline semantic-readiness services.

This repository is a software framework and comparative evaluation package, not a single neural checkpoint. config.json is its machine-readable project/model manifest; it is not a Transformers AutoConfig file.

Highlights

• Authoritative reference-text alignment with no silent ASR substitution.
• Reversible lexical mapping back to exact source transcript words.
• Strict finite, non-negative, monotonic, non-overlapping, WAV-bounded timestamps.
• No clamping, interpolation, extrapolation, or fabricated success/timing.
• Source-preserving, deterministic CSV annotation with atomic publication.
• Python-callable services; the CLI delegates to those services.
• Typed backend configuration, provenance, diagnostics, and optional native artifacts.
• Lazy backend isolation: missing NeMo/MFA/WhisperX dependencies do not break imports.
• Deterministic unit, architecture, header, manifest, and per-file coverage gates.

Model and Configuration Matrix

Backend	Pinned alignment model	Runtime baseline	Default device	Current role
MFA	MontrealCorpusTools/english_mfa@1009e3c6c689f10875146a5e4e5c2233f8eaf922 (english_us_mfa, model 3.3.0)	MFA 3.4.2	CPU	Selected operational backend
WhisperX	WAV2VEC2_ASR_BASE_960H / Torchaudio 2.8.0 bundle	WhisperX 3.8.6	CPU	Accepted alternative
NeMo / NFA	stt_en_fastconformer_hybrid_large_pc	NeMo 3.0.0 / source fd6a877539710e2b98f28c43272ff81312f83417	CUDA	Accepted; not selected operationally

All supported configuration fields, defaults, invariants, and JSON examples are in Configuration. Compact model, schema, and evaluation metadata are in config.json.

Installation

Use Python 3.12 or newer. Keep each heavyweight backend in its dedicated environment. The repository contains a deterministic development environment and separate backend environment specifications:

conda env create -f environments/core-dev.yml
conda activate speech-aligners-core-dev
python -m pip install -e .

An existing environment named speech-text-data-aligners is also suitable when it contains the packages pinned in requirements.txt:

conda deactivate
conda activate speech-text-data-aligners
python -m pip install -e . --no-deps

Backend environments are created separately from environments/mfa.yml, environments/nemo.yml, and environments/whisperx.yml. The parent CLI process invokes the selected isolated runtime; see Configuration.

Data Contract

Input metadata must contain these columns:

line_id,audio_id,en_transcript,ru_translation

Audio must be available at <audio_dir>/<audio_id>.wav. Annotation preserves every source header, cell value, CSV row, and row position, then appends exactly:

first_word_text,first_word_start_s,first_word_end_s,last_word_text,last_word_start_s,last_word_end_s

For a one-word utterance, first and last fields come from the same aligned word. Successful timing must satisfy:

0 <= first_start <= first_end <= last_start <= last_end <= audio_duration

Equality is allowed at backend resolution limits. NaN, infinity, negative values, reversed/overlapping intervals, incomplete lexical coverage, and out-of-audio values fail validation.

Python Usage

The root package exposes the stable backend-neutral API:

from pathlib import Path

from speech_text_data_aligners import AnnotationOptions, annotate_metadata
from speech_text_data_aligners.backends.mfa.backend import MfaBackend
from speech_text_data_aligners.backends.mfa.config import MfaConfig

run = annotate_metadata(
    metadata_csv=Path("metadata.csv"),
    audio_dir=Path("wavs"),
    output_csv=Path("annotated.csv"),
    backend=MfaBackend(MfaConfig(environment_name="speech-aligners-mfa")),
    options=AnnotationOptions(strict=False),
)

print(run.receipt.success_count, run.receipt.failure_count)

Custom or test backends can implement AlignmentBackend; application code receives them by dependency injection and does not branch on backend names.

For one backend:

from pathlib import Path

from speech_text_data_aligners import benchmark_backend
from speech_text_data_aligners.backends.mfa.backend import MfaBackend
from speech_text_data_aligners.backends.mfa.config import MfaConfig

report = benchmark_backend(
    backend=MfaBackend(MfaConfig()),
    metadata_csv=Path("metadata.csv"),
    reference_csv=Path("first-word-gold.csv"),
    audio_dir=Path("wavs"),
)

Use benchmark_backends for an insertion-ordered mapping of multiple backend instances. Specialized offline readiness services are documented in W1-T semantic readiness.

CLI Usage

After installation, speech-text-aligners and python -m speech_text_data_aligners.cli.main are equivalent entry points.

Annotate with the selected MFA environment:

speech-text-aligners annotate \
  metadata.csv wavs annotated.csv \
  --backend mfa \
  --backend-config-json '{"environment_name":"speech-aligners-mfa"}' \
  --receipt annotated.receipt.json

Benchmark multiple backends. Benchmark configuration is one JSON object keyed by the values passed to --backend:

speech-text-aligners benchmark \
  metadata.csv first-word-gold.csv wavs \
  --backend mfa \
  --backend whisperx \
  --backend-config-json '{"mfa":{"environment_name":"speech-aligners-mfa"},"whisperx":{"python_executable":"/path/to/whisperx/bin/python","device":"cpu"}}' \
  --output-base reports/comparison

Run offline semantic readiness with an explicit provider/evidence catalog:

speech-text-aligners readiness \
  source_metadata_en_only.csv boundaries.jsonl wavs results/w1t/new-attempt \
  --catalog /absolute/path/to/catalog.json

Run speech-text-aligners <command> --help for the authoritative argument list. Detailed workflows and output artifacts are in Workflows.

Failure and Publication Policy

• --strict aborts on the first row/backend failure and publishes no annotated CSV.
• Without --strict, failed rows retain their source cells, leave all six alignment cells blank, and are recorded in a deterministic diagnostics JSON file.
• Existing output artifacts require --overwrite.
• Annotation writes a complete temporary CSV, flushes and fsyncs it, then atomically publishes it. Input/output aliases are rejected before backend execution.
• Benchmarks retain structured failure observations and score only available references; failures never become fabricated timestamps.

Evaluation Summary

Manual gold exists for only first_word_text and first_word_end_s on five rows.

Backend	Success	Exact first-word text	First-word-end MAE	Median	Maximum
MFA	5/5	5/5	0.0162 s	0.0110 s	0.0300 s
WhisperX	5/5	5/5	0.0154 s	0.0140 s	0.0310 s
NeMo	5/5	5/5	0.2210 s	0.0100 s	0.9200 s

MFA and WhisperX are effectively tied on this sparse gold; the 0.0008 s MAE difference is not decision-grade. All three backends produced 36/36 structurally valid candidates. At a 0.250 s material-disagreement threshold, MFA/WhisperX differs on 7 rows, MFA/NeMo on 28, and WhisperX/NeMo on 32.

Only first-word text and first-word end have manual ground truth. First-word start and both last-word boundaries have structural validation and pairwise comparison, but no manual accuracy evidence. Pairwise agreement is diagnostic, not ground truth. Runtime observations used different hosts, devices, and cache states and are secondary.

See Evaluation for metric definitions, model/configuration coverage, full pairwise tables, W1-T evidence, caveats, and artifact paths.

Engineering Status

The current checkout passes 635 deterministic tests with 3 heavyweight integration tests skipped, 98.74% aggregate line coverage, all individual implementation-file coverage gates, Ruff, strict mypy, and all 7 import-linter contracts. The current lowest implementation file is 93.33%. Commands and interpretation are in Testing and Quality.

Documentation

Current guidance:

• Architecture
• Configuration
• Evaluation
• Workflows
• Testing and Quality
• Project Status
• Documentation Map

Contracts and evidence:

• Aligner Engineering Contract
• Modular Architecture Contract
• Final Aligner Selection
• Post-final Three-Aligner Comparison
• W1-T Semantic Readiness

Historical stage reports remain unchanged as implementation and acceptance provenance; they are not the primary operating guide.

License

Licensed under the Apache License 2.0.

Copyright (c) 2026 VertoX-AI. All rights reserved.

Author

Patrick Lumbantobing - VertoX-AI

Citation

@misc{vertoxai2026speechtextdataaligners,
  title        = {Speech-Text Data Aligners},
  author       = {Tobing, P. L. and VertoX-AI},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/vertox-ai/speech-text-data-aligners}}
}

References

• Hugging Face model cards
• NVIDIA NeMo Forced Aligner
• Montreal Forced Aligner
• WhisperX
• PyTorch
• Torchaudio WAV2VEC2_ASR_BASE_960H

Acknowledgment does not imply endorsement by any upstream project or organization.