vertox-ai/speech-text-data-aligners

Speech-Text Data Aligners

A library-first framework that uses a known, authoritative English transcript to estimate word timing in WAV audio. It performs forced alignment, not ASR transcription, and never replaces en_transcript with recognized text.

Features

• One backend-neutral Python contract for NeMo, MFA, and WhisperX.
• Reversible normalization and monotonic mapping back to original transcript words.
• Validated, non-fabricated timestamps with structured failures and provenance.
• Source-preserving, deterministic CSV annotation with atomic publication.
• Reusable annotation and cross-backend benchmark services independent of the CLI.
• Lazy, isolated optional backends; ordinary tests require no model, GPU, or network.

Supported Backends

Backend	Implementation	Empirical status
NVIDIA NeMo Forced Aligner (NFA)	Accepted	Real NVIDIA A10G smoke, five-row benchmark, and strict 36-row candidate passed
Montreal Forced Aligner (MFA) 3.x	Accepted	Selected final backend; five-row and strict full-36 validation passed
WhisperX 3.8.6	Accepted	Five-row and strict full-36 validation passed; accepted alternative

Implementation acceptance and empirical validation are separate. See Project Status for the evidence boundary.

Input and Output

Input metadata must contain:

line_id,audio_id,en_transcript,ru_translation

Audio is resolved as <audio_dir>/<audio_id>.wav. Annotation preserves all source columns, values, and row order, then appends exactly:

first_word_text,first_word_start_s,first_word_end_s,
last_word_text,last_word_start_s,last_word_end_s

en_transcript is authoritative. Backends may create internal reference representations, but independent ASR output cannot become the transcript.

Quick Start

Use the pinned environment definitions in environments/ and keep backend runtimes isolated. Detailed setup and execution paths are in Workflows.

Minimal Python API:

from pathlib import Path

from speech_text_data_aligners import AnnotationOptions, annotate_metadata
from speech_text_data_aligners.backends.mfa.backend import MfaBackend
from speech_text_data_aligners.backends.mfa.config import MfaConfig

run = annotate_metadata(
    metadata_csv=Path("metadata.csv"),
    audio_dir=Path("wavs"),
    output_csv=Path("annotated.csv"),
    backend=MfaBackend(MfaConfig(environment_name="speech-aligners-mfa")),
    options=AnnotationOptions(strict=False),
)
print(run.receipt.success_count, run.receipt.failure_count)

Minimal CLI:

PYTHONPATH=src python -m speech_text_data_aligners.cli.main annotate \
  metadata.csv wavs annotated.csv \
  --backend mfa \
  --backend-config-json '{"environment_name":"speech-aligners-mfa"}'

Use --strict to abort publication on the first failure. Without it, failed source rows are preserved with six blank alignment cells and a diagnostics JSON artifact.

Documentation

• Project status
• Architecture
• Workflows
• Aligner engineering contract
• Modular architecture contract
• Stage 5 final acceptance report
• Post-final three-aligner comparison
• Final aligner selection
• W1-T offline semantic-readiness timing

Current Validation Status

Stages 0–5 are accepted, the NVIDIA-host NeMo closure is complete, and the post-final MFA extension has closed the full-corpus comparison. All three backends now have successful 36/36 candidates. On the five shared manual-gold rows, first-word-end MAE is 0.0162 s for MFA, 0.0154 s for WhisperX, and 0.2210 s for NeMo.

MFA is the selected operational backend after the symmetric 36-row comparison. The earlier WhisperX selection was made before MFA had a strict full-36 result and is superseded by the 2026-09-06 post-final review. The canonical output is results/final/final-annotated-36.csv (36/36 rows; SHA-256 37bcb85fc22fb142dae40453b90dd3b427bbfd657861bee7529cd433915535a8).

Only first-word text and first-word end currently have manual gold. First-word start, last-word start, and last-word end have structural validation and pairwise comparison, but no manual ground-truth validation. Pairwise agreement remains review evidence, not ground truth.

License & Copyright

Licensed under the Apache License 2.0. The official license text is also available from the Apache Software Foundation.

Copyright (c) 2026 VertoX-AI. All rights reserved.

Author

Patrick Lumbantobing — VertoX-AI

Citation

@misc{vertoxai2026speechtextdataaligners,
  title        = {Speech-Text Data Aligners},
  author       = {Tobing, P. L. and VertoX-AI},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/vertox-ai/speech-text-data-aligners}}
}

Acknowledgments

This project builds on NVIDIA NeMo and NeMo Forced Aligner, Montreal Forced Aligner, WhisperX, PyTorch, Torchaudio, and Hugging Face. WhisperX uses Torchaudio's selected WAV2VEC2_ASR_BASE_960H bundle. Acknowledgment does not imply endorsement by any upstream project or organization.