NeMo
English
audio
speech
forced-alignment
reference-text-alignment
speech-processing
mfa
whisperx
python
metric:mae
Eval Results (legacy)
Instructions to use vertox-ai/speech-text-data-aligners with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- NeMo
How to use vertox-ai/speech-text-data-aligners with NeMo:
# tag did not correspond to a valid NeMo domain.
- Notebooks
- Google Colab
- Kaggle
|
Download ARCHITECTURE.md from vertox-ai/speech-text-data-aligners: direct link, hf CLI and curl.
- Browser
- Download file 11.7 kB
-
https://huggingface.co/vertox-ai/speech-text-data-aligners/resolve/main/ARCHITECTURE.md
- Command line
-
hf download hf://vertox-ai/speech-text-data-aligners/ARCHITECTURE.md
-
curl -L -o ARCHITECTURE.md https://huggingface.co/vertox-ai/speech-text-data-aligners/resolve/main/ARCHITECTURE.md
11.7 kB
| # Architecture | |
| Speech-Text Data Aligners is a library-first forced-alignment framework. The core | |
| contract accepts WAV audio plus a known, authoritative transcript and returns ordered | |
| backend-neutral timed words. The CLI is a composition and presentation layer. | |
| The mandatory design rules are in | |
| [`docs/MODULAR_ARCHITECTURE_CONTRACT.md`](docs/MODULAR_ARCHITECTURE_CONTRACT.md). | |
| Runtime alignment semantics are in | |
| [`docs/ALIGNER_ENGINEERING_CONTRACT.md`](docs/ALIGNER_ENGINEERING_CONTRACT.md). | |
| ## Design Goals | |
| - Keep authoritative reference text immutable. | |
| - Keep core logic deterministic and independent of model/runtime packages. | |
| - Make MFA, NeMo, and WhisperX interchangeable through `AlignmentBackend`. | |
| - Contain native formats inside their adapters. | |
| - Provide reusable Python services rather than CLI-only workflows. | |
| - Keep optional runtimes independently unavailable. | |
| - Preserve source metadata and deterministic output order. | |
| - Represent failure explicitly and never fabricate timing. | |
| ## Layer and Dependency Direction | |
| ```text | |
| CLI --------------------> Application --------------------> Core | |
| | ^ | |
| +----------> Data -------------+ | |
| +----------> Benchmarking -----+ | |
| +----------> Runtime ----------+ | |
| CLI -- lazy composition --> Concrete Backends | |
| | | | |
| +------------> Core | |
| +------------> Runtime | |
| ``` | |
| Enforced consequences: | |
| - Core does not import application, data, benchmarking, runtime, CLI, or backends. | |
| - Data and benchmarking do not import concrete backends or CLI. | |
| - Application services receive `AlignmentBackend` implementations by injection and do | |
| not branch on concrete backend names. | |
| - Concrete backend packages may depend on core/runtime and their own native modules, | |
| but never on a sibling backend. | |
| - CLI composition may select a registry factory; it does not own alignment logic. | |
| - Backend factories are lazy dotted references, so importing the public library does | |
| not load heavyweight aligner dependencies. | |
| ## Current Package Tree | |
| ```text | |
| src/speech_text_data_aligners/ | |
| |-- __init__.py stable public facade | |
| |-- core/ | |
| | |-- models.py immutable neutral request/result/provenance values | |
| | |-- protocols.py AlignmentBackend and structural contracts | |
| | |-- errors.py project exception/failure taxonomy | |
| | |-- normalization.py lexical plans and monotonic reconciliation | |
| | |-- validation.py timing, lexical, and publication-path invariants | |
| | |-- extraction.py first/last word output fields | |
| | `-- readiness.py semantic-boundary mapping and consensus | |
| |-- data/ | |
| | |-- metadata.py preserving, atomic CSV input/output | |
| | |-- audio.py audio resolution and lightweight WAV probing | |
| | `-- readiness.py frozen readiness cohort/hash verification | |
| |-- application/ | |
| | |-- annotate.py reusable annotation orchestration | |
| | |-- benchmark.py identical-request benchmark orchestration | |
| | |-- readiness.py evidence reuse and offline readiness execution | |
| | `-- materialize.py accepted-evidence full-word timing materialization | |
| |-- benchmarking/ | |
| | |-- metrics.py sparse-gold, runtime, and pairwise metrics | |
| | |-- reports.py deterministic JSON/CSV/Markdown reporting | |
| | `-- readiness.py readiness distributions and sensitivity summaries | |
| |-- runtime/ | |
| | |-- process.py shell-free injectable subprocess boundary | |
| | |-- artifacts.py temporary/retained artifact lifecycle | |
| | |-- receipts.py hashes, host facts, and run receipts | |
| | `-- alignment_evidence.py full-word evidence integrity and verification | |
| |-- backends/ | |
| | |-- registry.py immutable lazy backend factories | |
| | |-- nemo/ | |
| | | |-- config.py typed NFA runtime/model configuration | |
| | | |-- manifest.py reference/output manifest parsing | |
| | | |-- runner.py isolated NFA execution | |
| | | |-- ctm.py CTM parsing/mapping | |
| | | `-- backend.py neutral NFA adapter | |
| | |-- mfa/ | |
| | | |-- config.py typed MFA runtime/model configuration | |
| | | |-- corpus.py temporary corpus/label construction | |
| | | |-- runner.py isolated MFA execution | |
| | | |-- textgrid.py TextGrid parsing/mapping | |
| | | `-- backend.py neutral MFA adapter | |
| | `-- whisperx/ | |
| | |-- config.py typed worker/runtime/model configuration | |
| | |-- segments.py full-duration reference segments | |
| | |-- provider.py isolated-worker transport | |
| | |-- worker.py lazy heavyweight native runtime | |
| | |-- parser.py native word parsing/mapping | |
| | `-- backend.py neutral WhisperX adapter | |
| `-- cli/ | |
| |-- main.py command dispatch and typed-error exit status | |
| |-- annotate.py annotation argument adapter | |
| |-- benchmark.py benchmark argument adapter | |
| `-- readiness.py readiness catalog/composition adapter | |
| ``` | |
| Empty package-marker `__init__.py` files are deliberately zero bytes. Each non-empty | |
| implementation module has a focused test module and meets its individual line-coverage | |
| gate. | |
| ## Module Responsibilities | |
| | Area | Owns | Does not own | | |
| |---|---|---| | |
| | `core` | Neutral values, normalization/reconciliation, validation, extraction, readiness rules | CSV, subprocesses, native formats, model loading | | |
| | `data` | Metadata/audio/cohort input and source preservation | Backend selection or metrics | | |
| | `application` | Reuse-case orchestration and publication policy | Native parsing or benchmark math | | |
| | `benchmarking` | Deterministic metrics and report rendering | Backend execution or native result dictionaries | | |
| | `runtime` | Process, artifact, receipt, and evidence integrity primitives | Application row loops or model semantics | | |
| | `backends.*` | Config, native inputs/outputs, execution, parsing, mapping | Metadata orchestration or sibling backend imports | | |
| | `cli` | Argument parsing, backend composition, presentation | Alignment, CSV loops, metric computation | | |
| ## Shared Alignment Contract | |
| ```python | |
| result = backend.align(request) | |
| outcomes = backend.align_batch(requests) | |
| ``` | |
| `AlignmentRequest` contains: | |
| - caller/request identity; | |
| - resolved WAV path; | |
| - exact authoritative transcript; | |
| - probed WAV duration; | |
| - immutable JSON-safe context. | |
| A successful `AlignmentResult` contains one `WordAlignment` per original lexical word, | |
| plus provenance, diagnostics, and measured runtime. A failure is represented by | |
| `AlignmentFailure`; it contains no timing fields. `AlignmentOutcome` enforces exactly | |
| one result/failure variant. | |
| The root package exports `AlignmentBackend`, neutral request/result/failure/provenance | |
| models, annotation services, and benchmark services. Custom backends can implement the | |
| protocol without importing a concrete adapter. | |
| ## Transcript and Alignment Flow | |
| ```text | |
| source metadata | |
| | | |
| +-- immutable en_transcript | |
| | | |
| +-- resolve <audio_dir>/<audio_id>.wav and probe duration | |
| v | |
| lexical preflight and original-character mapping | |
| v | |
| backend-specific reversible reference representation | |
| v | |
| native model/process execution | |
| v | |
| backend-local native output parsing | |
| v | |
| shared monotonic reconciliation and validation | |
| v | |
| WordAlignment / AlignmentFailure | |
| v | |
| first/last extraction or benchmark observation | |
| ``` | |
| Backends may normalize punctuation, case, numbers, abbreviations, or native token | |
| units internally. Every successful native token must map deterministically to the | |
| original transcript's lexical words. Shared validation rejects incomplete coverage, | |
| non-finite/negative/reversed/overlapping timing, and out-of-audio values. It never | |
| clamps, interpolates, extrapolates, or invents a timestamp. | |
| ## Native Boundary Containment | |
| | Backend | Native structures contained locally | External boundary | | |
| |---|---|---| | |
| | NeMo | Reference/output manifests and word CTM rows | Isolated NFA process | | |
| | MFA | Temporary WAV/LAB corpus and TextGrid tiers | Isolated Conda MFA process | | |
| | WhisperX | Reference segments and word-segment dictionaries | Isolated heavyweight worker | | |
| Only neutral models, diagnostics, and provenance cross these boundaries. Parent-process | |
| imports do not require NeMo, MFA, WhisperX, Torch, or Torchaudio. | |
| ## Annotation Publication | |
| `application.annotate` prepares all source rows, invokes one injected backend, validates | |
| and merges outcomes in source order, extracts six boundary columns, and atomically | |
| publishes the enriched CSV. Continue-on-error preserves failed source rows with blank | |
| alignment cells and structured diagnostics. Strict mode publishes no annotated output | |
| after a row failure. | |
| ## Benchmark Composition | |
| `application.benchmark` constructs one immutable ordered request tuple and supplies it | |
| to every injected backend. It joins sparse manual gold by `(line_id, audio_id)`, scores | |
| available first-word text/end fields, and compares successful first/last boundaries | |
| pairwise. Reports are backend-neutral; native artifact paths appear only as opaque | |
| references. | |
| ## Offline Readiness and Evidence | |
| The readiness extension retains the same backend isolation: | |
| ```text | |
| metadata + frozen boundaries + WAVs | |
| | | |
| v | |
| cohort/hash verification | |
| | | |
| +-- verified full-word evidence capsules --> reuse audit | |
| | | |
| `-- missing/rejected case-backend pairs --> injected AlignmentBackend | |
| | | |
| v | |
| full-word evidence verification | |
| | | |
| v | |
| boundary mapping + consensus/reporting | |
| ``` | |
| `core.readiness` owns mapping/extraction/consensus, `data.readiness` owns cohort | |
| verification, `runtime.alignment_evidence` owns capsule integrity, and | |
| `application.readiness` orchestrates reuse/execution. `application.materialize` | |
| materializes deterministic full-word timing from accepted immutable evidence without | |
| importing or executing an aligner. | |
| ## Optional Dependency and Registry Behavior | |
| `BackendRegistry` stores immutable dotted factories. Construction imports only the | |
| selected backend module and validates the result against `AlignmentBackend`. An absent | |
| optional runtime raises an actionable `BackendUnavailableError` only when selected. | |
| One backend's absence does not affect backend-neutral imports or sibling construction. | |
| ## Architecture Verification | |
| Release gates enforce: | |
| - AST/import-boundary tests for prohibited directions and heavyweight imports; | |
| - seven Import Linter contracts; | |
| - backend sibling isolation; | |
| - public-library tests with fake/injected backends; | |
| - optional dependency absence; | |
| - focused implementation-to-test mapping; | |
| - per-file and aggregate coverage. | |
| The current observed gate is documented in [`docs/TESTING.md`](docs/TESTING.md). | |
| ## Evolution Rules | |
| Adding a backend should require a config, native input/execution/parser adapter, and | |
| focused tests. It should not require changes to metadata I/O, common normalization, | |
| first/last extraction, benchmark metrics, or the CLI beyond registry composition. | |
| Shared helpers are introduced only when semantics are genuinely identical. | |