Hugging Face's logo

vertox-ai
/
speech-text-data-aligners

NeMo
English
audio
speech
forced-alignment
reference-text-alignment
speech-processing
mfa
whisperx
python
metric:mae
Eval Results (legacy)
Model card Files
xet
 Community
speech-text-data-aligners / ARCHITECTURE.md
pltobing's picture
pltobing
docs: refine aligner documentation
a3b146e
preview code
|
Raw History Blame Contribute Delete
11.7 kB

Architecture

Speech-Text Data Aligners is a library-first forced-alignment framework. The core contract accepts WAV audio plus a known, authoritative transcript and returns ordered backend-neutral timed words. The CLI is a composition and presentation layer.

The mandatory design rules are in docs/MODULAR_ARCHITECTURE_CONTRACT.md. Runtime alignment semantics are in docs/ALIGNER_ENGINEERING_CONTRACT.md.

Design Goals

  • Keep authoritative reference text immutable.
  • Keep core logic deterministic and independent of model/runtime packages.
  • Make MFA, NeMo, and WhisperX interchangeable through AlignmentBackend.
  • Contain native formats inside their adapters.
  • Provide reusable Python services rather than CLI-only workflows.
  • Keep optional runtimes independently unavailable.
  • Preserve source metadata and deterministic output order.
  • Represent failure explicitly and never fabricate timing.

Layer and Dependency Direction 

CLI --------------------> Application --------------------> Core
                              |                              ^
                              +----------> Data -------------+
                              +----------> Benchmarking -----+
                              +----------> Runtime ----------+

CLI -- lazy composition --> Concrete Backends
                            |              |
                            +------------> Core
                            +------------> Runtime

Enforced consequences:

  • Core does not import application, data, benchmarking, runtime, CLI, or backends.
  • Data and benchmarking do not import concrete backends or CLI.
  • Application services receive AlignmentBackend implementations by injection and do not branch on concrete backend names.
  • Concrete backend packages may depend on core/runtime and their own native modules, but never on a sibling backend.
  • CLI composition may select a registry factory; it does not own alignment logic.
  • Backend factories are lazy dotted references, so importing the public library does not load heavyweight aligner dependencies.

Current Package Tree 

src/speech_text_data_aligners/
|-- __init__.py                    stable public facade
|-- core/
|   |-- models.py                  immutable neutral request/result/provenance values
|   |-- protocols.py               AlignmentBackend and structural contracts
|   |-- errors.py                  project exception/failure taxonomy
|   |-- normalization.py           lexical plans and monotonic reconciliation
|   |-- validation.py              timing, lexical, and publication-path invariants
|   |-- extraction.py              first/last word output fields
|   `-- readiness.py               semantic-boundary mapping and consensus
|-- data/
|   |-- metadata.py                preserving, atomic CSV input/output
|   |-- audio.py                   audio resolution and lightweight WAV probing
|   `-- readiness.py               frozen readiness cohort/hash verification
|-- application/
|   |-- annotate.py                reusable annotation orchestration
|   |-- benchmark.py               identical-request benchmark orchestration
|   |-- readiness.py               evidence reuse and offline readiness execution
|   `-- materialize.py             accepted-evidence full-word timing materialization
|-- benchmarking/
|   |-- metrics.py                 sparse-gold, runtime, and pairwise metrics
|   |-- reports.py                 deterministic JSON/CSV/Markdown reporting
|   `-- readiness.py               readiness distributions and sensitivity summaries
|-- runtime/
|   |-- process.py                 shell-free injectable subprocess boundary
|   |-- artifacts.py               temporary/retained artifact lifecycle
|   |-- receipts.py                hashes, host facts, and run receipts
|   `-- alignment_evidence.py      full-word evidence integrity and verification
|-- backends/
|   |-- registry.py                immutable lazy backend factories
|   |-- nemo/
|   |   |-- config.py              typed NFA runtime/model configuration
|   |   |-- manifest.py            reference/output manifest parsing
|   |   |-- runner.py              isolated NFA execution
|   |   |-- ctm.py                 CTM parsing/mapping
|   |   `-- backend.py             neutral NFA adapter
|   |-- mfa/
|   |   |-- config.py              typed MFA runtime/model configuration
|   |   |-- corpus.py              temporary corpus/label construction
|   |   |-- runner.py              isolated MFA execution
|   |   |-- textgrid.py            TextGrid parsing/mapping
|   |   `-- backend.py             neutral MFA adapter
|   `-- whisperx/
|       |-- config.py              typed worker/runtime/model configuration
|       |-- segments.py            full-duration reference segments
|       |-- provider.py            isolated-worker transport
|       |-- worker.py              lazy heavyweight native runtime
|       |-- parser.py              native word parsing/mapping
|       `-- backend.py             neutral WhisperX adapter
`-- cli/
    |-- main.py                    command dispatch and typed-error exit status
    |-- annotate.py                annotation argument adapter
    |-- benchmark.py               benchmark argument adapter
    `-- readiness.py               readiness catalog/composition adapter

Empty package-marker __init__.py files are deliberately zero bytes. Each non-empty implementation module has a focused test module and meets its individual line-coverage gate.

Module Responsibilities

Area Owns Does not own
core Neutral values, normalization/reconciliation, validation, extraction, readiness rules CSV, subprocesses, native formats, model loading
data Metadata/audio/cohort input and source preservation Backend selection or metrics
application Reuse-case orchestration and publication policy Native parsing or benchmark math
benchmarking Deterministic metrics and report rendering Backend execution or native result dictionaries
runtime Process, artifact, receipt, and evidence integrity primitives Application row loops or model semantics
backends.* Config, native inputs/outputs, execution, parsing, mapping Metadata orchestration or sibling backend imports
cli Argument parsing, backend composition, presentation Alignment, CSV loops, metric computation

Shared Alignment Contract 

result = backend.align(request)
outcomes = backend.align_batch(requests)

AlignmentRequest contains:

  • caller/request identity;
  • resolved WAV path;
  • exact authoritative transcript;
  • probed WAV duration;
  • immutable JSON-safe context.

A successful AlignmentResult contains one WordAlignment per original lexical word, plus provenance, diagnostics, and measured runtime. A failure is represented by AlignmentFailure; it contains no timing fields. AlignmentOutcome enforces exactly one result/failure variant.

The root package exports AlignmentBackend, neutral request/result/failure/provenance models, annotation services, and benchmark services. Custom backends can implement the protocol without importing a concrete adapter.

Transcript and Alignment Flow 

source metadata
    |
    +-- immutable en_transcript
    |
    +-- resolve <audio_dir>/<audio_id>.wav and probe duration
    v
lexical preflight and original-character mapping
    v
backend-specific reversible reference representation
    v
native model/process execution
    v
backend-local native output parsing
    v
shared monotonic reconciliation and validation
    v
WordAlignment / AlignmentFailure
    v
first/last extraction or benchmark observation

Backends may normalize punctuation, case, numbers, abbreviations, or native token units internally. Every successful native token must map deterministically to the original transcript's lexical words. Shared validation rejects incomplete coverage, non-finite/negative/reversed/overlapping timing, and out-of-audio values. It never clamps, interpolates, extrapolates, or invents a timestamp.

Native Boundary Containment

Backend Native structures contained locally External boundary
NeMo Reference/output manifests and word CTM rows Isolated NFA process
MFA Temporary WAV/LAB corpus and TextGrid tiers Isolated Conda MFA process
WhisperX Reference segments and word-segment dictionaries Isolated heavyweight worker

Only neutral models, diagnostics, and provenance cross these boundaries. Parent-process imports do not require NeMo, MFA, WhisperX, Torch, or Torchaudio.

Annotation Publication

application.annotate prepares all source rows, invokes one injected backend, validates and merges outcomes in source order, extracts six boundary columns, and atomically publishes the enriched CSV. Continue-on-error preserves failed source rows with blank alignment cells and structured diagnostics. Strict mode publishes no annotated output after a row failure.

Benchmark Composition

application.benchmark constructs one immutable ordered request tuple and supplies it to every injected backend. It joins sparse manual gold by (line_id, audio_id), scores available first-word text/end fields, and compares successful first/last boundaries pairwise. Reports are backend-neutral; native artifact paths appear only as opaque references.

Offline Readiness and Evidence

The readiness extension retains the same backend isolation:

metadata + frozen boundaries + WAVs
    |
    v
cohort/hash verification
    |
    +-- verified full-word evidence capsules --> reuse audit
    |
    `-- missing/rejected case-backend pairs --> injected AlignmentBackend
                                                  |
                                                  v
                                      full-word evidence verification
                                                  |
                                                  v
                                  boundary mapping + consensus/reporting

core.readiness owns mapping/extraction/consensus, data.readiness owns cohort verification, runtime.alignment_evidence owns capsule integrity, and application.readiness orchestrates reuse/execution. application.materialize materializes deterministic full-word timing from accepted immutable evidence without importing or executing an aligner.

Optional Dependency and Registry Behavior

BackendRegistry stores immutable dotted factories. Construction imports only the selected backend module and validates the result against AlignmentBackend. An absent optional runtime raises an actionable BackendUnavailableError only when selected. One backend's absence does not affect backend-neutral imports or sibling construction.

Architecture Verification

Release gates enforce:

  • AST/import-boundary tests for prohibited directions and heavyweight imports;
  • seven Import Linter contracts;
  • backend sibling isolation;
  • public-library tests with fake/injected backends;
  • optional dependency absence;
  • focused implementation-to-test mapping;
  • per-file and aggregate coverage.

The current observed gate is documented in docs/TESTING.md.

Evolution Rules

Adding a backend should require a config, native input/execution/parser adapter, and focused tests. It should not require changes to metadata I/O, common normalization, first/last extraction, benchmark metrics, or the CLI beyond registry composition. Shared helpers are introduced only when semantics are genuinely identical.