Instructions to use vertox-ai/speech-text-data-aligners with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- NeMo
How to use vertox-ai/speech-text-data-aligners with NeMo:
# tag did not correspond to a valid NeMo domain.
- Notebooks
- Google Colab
- Kaggle
docs: refine aligner documentation
Browse files- Rebuild README.md as a Hugging Face-style framework card with tags, license, model/configuration coverage, usage, metrics, limitations, and references.
- Add config.json as a tested machine-readable project/model manifest and bind its schema, versions, backends, model identities, and canonical hash to source/evidence values.
- Add root ARCHITECTURE.md and focused configuration, evaluation, testing, workflow, and documentation-map guidance for current operating use.
- Preserve historical reports while clarifying current contracts, empirical boundaries, readiness outputs, and the selected MFA operational configuration.
- Add deterministic manifest and local Markdown-link quality tests with approved Python headers.
- Record plans, review drafts, baseline/final work data, and validation logs under .agents while excluding private and raw orchestrator artifacts.
- Update AGENTS.md and SKILLS.md with durable documentation and manifest integrity requirements.
- .agents/drafts/documentation-refinement-review.md +67 -0
- .agents/logs/documentation-refinement-20261030T152745Z.md +43 -0
- .agents/plans/01-review-and-documentation-refinement.md +63 -0
- .agents/work_data/documentation-final-verification.md +24 -0
- .agents/work_data/documentation-review-baseline.md +72 -0
- .gitignore +7 -0
- AGENTS.md +7 -0
- ARCHITECTURE.md +259 -0
- README.md +228 -70
- SKILLS.md +16 -0
- config.json +162 -0
- docs/ARCHITECTURE.md +3 -185
- docs/CONFIGURATION.md +282 -0
- docs/DOCUMENTATION.md +87 -0
- docs/EVALUATION.md +195 -0
- docs/MODULAR_ARCHITECTURE_CONTRACT.md +20 -6
- docs/PROJECT_STATUS.md +22 -0
- docs/TESTING.md +177 -0
- docs/W1_T_SEMANTIC_READINESS.md +4 -0
- docs/WORKFLOWS.md +71 -0
- tests/quality/test_config_manifest.py +76 -0
- tests/quality/test_documentation_links.py +67 -0
|
@@ -0,0 +1,67 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Documentation Refinement Review Draft
|
| 2 |
+
|
| 3 |
+
Date: 2026-10-30
|
| 4 |
+
|
| 5 |
+
## Purpose
|
| 6 |
+
|
| 7 |
+
Turn the repository's implementation, contracts, configuration, and empirical evidence
|
| 8 |
+
into a concise documentation set that is accurate enough to use without reading the
|
| 9 |
+
historical stage reports.
|
| 10 |
+
|
| 11 |
+
## Baseline findings
|
| 12 |
+
|
| 13 |
+
- The library-first architecture and authoritative transcript contract are implemented
|
| 14 |
+
and enforced by deterministic tests and seven import-linter contracts.
|
| 15 |
+
- The README has useful model-card elements but lacks complete tags, an explicit
|
| 16 |
+
configuration matrix, metric definitions, evaluation-table caveats, and a clear
|
| 17 |
+
distinction between current guidance and archival reports.
|
| 18 |
+
- `docs/ARCHITECTURE.md` omits the current readiness, evidence, and materialization
|
| 19 |
+
modules from its package tree and layer narrative.
|
| 20 |
+
- `docs/WORKFLOWS.md` covers annotation and benchmarking well but does not provide one
|
| 21 |
+
authoritative CLI/API configuration reference or report artifact contract.
|
| 22 |
+
- Backend defaults, model identifiers, immutable revisions, runtime versions, artifact
|
| 23 |
+
policies, and required path fields are spread across source docstrings and reports.
|
| 24 |
+
- No root machine-readable configuration manifest currently complements the model-card
|
| 25 |
+
README; the requested `config.json` should record canonical project/model metadata and
|
| 26 |
+
pinned backend/model identities without pretending to be a Transformers checkpoint.
|
| 27 |
+
- Evaluation numbers are accurate but need one canonical comparison page that states
|
| 28 |
+
sample size, supported gold fields, aggregation rules, runtime caveats, and the
|
| 29 |
+
boundary between accuracy and pairwise disagreement.
|
| 30 |
+
- Existing stage reports are valuable provenance but should be labeled as historical
|
| 31 |
+
evidence rather than current operating instructions.
|
| 32 |
+
- AGENTS.md and SKILLS.md are accurate for implementation work. The documentation task
|
| 33 |
+
should add review/verification workflow guidance without weakening existing gates.
|
| 34 |
+
|
| 35 |
+
## Planned documentation changes
|
| 36 |
+
|
| 37 |
+
1. Record the review baseline and per-file coverage evidence under `.agents/work_data/`.
|
| 38 |
+
2. Add `config.json` as a versioned machine-readable project/model manifest.
|
| 39 |
+
3. Rewrite README.md as the Hugging Face-style entry point with model metadata, tags,
|
| 40 |
+
supported models/configurations, quick starts, metrics, limitations, and references.
|
| 41 |
+
4. Expand the architecture documentation to include readiness/evidence/materialization
|
| 42 |
+
responsibilities and current dependency boundaries.
|
| 43 |
+
5. Add focused configuration, evaluation, and testing/quality references.
|
| 44 |
+
6. Refine workflows and project status so commands, outputs, exit behavior, and evidence
|
| 45 |
+
paths are easy to locate.
|
| 46 |
+
7. Add a documentation map that separates current contracts from historical reports.
|
| 47 |
+
8. Validate links, CLI help, deterministic tests, lint/type/import gates, coverage, and
|
| 48 |
+
Python header policy; record results in `.agents/logs/`.
|
| 49 |
+
9. Review AGENTS.md and SKILLS.md again and make only necessary documentation-quality
|
| 50 |
+
updates.
|
| 51 |
+
|
| 52 |
+
## Non-goals
|
| 53 |
+
|
| 54 |
+
- Do not change source metadata, translations, canonical result CSVs, or empirical
|
| 55 |
+
evidence artifacts.
|
| 56 |
+
- Do not claim manual accuracy for unannotated boundaries.
|
| 57 |
+
- Do not add implementation scripts solely for documentation generation.
|
| 58 |
+
- Do not push to a remote repository.
|
| 59 |
+
|
| 60 |
+
## Finalized outcome
|
| 61 |
+
|
| 62 |
+
- Added a Hugging Face-style README and machine-readable `config.json` manifest.
|
| 63 |
+
- Added focused architecture, configuration, evaluation, workflow, testing, and
|
| 64 |
+
documentation-map guides.
|
| 65 |
+
- Added deterministic manifest and local-link quality tests.
|
| 66 |
+
- Updated AGENTS.md/SKILLS.md with documentation and manifest integrity guidance.
|
| 67 |
+
- Preserved historical evidence and source metadata unchanged.
|
|
@@ -0,0 +1,43 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Documentation Refinement Log
|
| 2 |
+
|
| 3 |
+
## 2026-10-30T15:27:45Z - Start
|
| 4 |
+
|
| 5 |
+
- Read AGENTS.md, SKILLS.md, the aligner engineering contract, and the modular
|
| 6 |
+
architecture contract before implementation.
|
| 7 |
+
- Inventoried tracked source, tests, environments, documentation, results, and untracked
|
| 8 |
+
`.agents/` materials.
|
| 9 |
+
- Compared README and current docs with public exports, CLI help, typed backend configs,
|
| 10 |
+
application services, metrics/report schemas, readiness flow, and environment specs.
|
| 11 |
+
- Recorded baseline deterministic evidence: 633 passed / 3 skipped, 98.74% aggregate
|
| 12 |
+
coverage, lowest file 93.33%, Ruff pass, strict mypy pass, 7 import contracts kept.
|
| 13 |
+
- Created the requested review draft, execution plan, and baseline work data before
|
| 14 |
+
changing user-facing documentation.
|
| 15 |
+
|
| 16 |
+
## 2026-10-30 - Documentation implementation
|
| 17 |
+
|
| 18 |
+
- Preserved the user-authored `.gitignore` exclusions for `.agents/config`,
|
| 19 |
+
`.agents/user_prompts`, and `.agents/user_scripts`.
|
| 20 |
+
- Added root `config.json` as a versioned project/model manifest and tested its schema,
|
| 21 |
+
registry/model identity, package version, and canonical result hash.
|
| 22 |
+
- Reworked README.md into a Hugging Face-style model/framework card with tags, license,
|
| 23 |
+
model/configuration comparison, usage, metrics, limitations, and references.
|
| 24 |
+
- Added root ARCHITECTURE.md plus focused CONFIGURATION.md, EVALUATION.md, TESTING.md,
|
| 25 |
+
and DOCUMENTATION.md guides; refined WORKFLOWS.md, PROJECT_STATUS.md, the architecture
|
| 26 |
+
contract, and W1-T provenance labeling.
|
| 27 |
+
- Added deterministic local Markdown-link validation and finalized AGENTS.md/SKILLS.md
|
| 28 |
+
documentation-integrity guidance.
|
| 29 |
+
|
| 30 |
+
## 2026-10-30 - Final verification
|
| 31 |
+
|
| 32 |
+
- `python -m pytest -q`: 635 passed, 3 skipped.
|
| 33 |
+
- `coverage run -m pytest -q`: 635 passed, 3 skipped.
|
| 34 |
+
- `coverage report --format=total`: 98.74%; lowest implementation file 93.33%.
|
| 35 |
+
- `python -m ruff check .`: pass.
|
| 36 |
+
- `python -m ruff format --check .`: 150 files already formatted.
|
| 37 |
+
- `python -m mypy src`: pass across 53 source files.
|
| 38 |
+
- `lint-imports`: 7 contracts kept, 0 broken.
|
| 39 |
+
- Architecture tests: 10 passed.
|
| 40 |
+
- Header, manifest, documentation-link, and per-file coverage tests: pass.
|
| 41 |
+
- `speech-text-aligners` and all three subcommand help surfaces: pass.
|
| 42 |
+
- `conda env create --help`, `python -m pip install --help`, and the named Conda
|
| 43 |
+
environment Python check: pass.
|
|
@@ -0,0 +1,63 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Plan: Repository Review and Documentation Refinement
|
| 2 |
+
|
| 3 |
+
Status: complete
|
| 4 |
+
|
| 5 |
+
## Objective
|
| 6 |
+
|
| 7 |
+
Produce a comprehensive, navigable, and reproducible documentation set for using,
|
| 8 |
+
configuring, extending, evaluating, and verifying Speech-Text Data Aligners.
|
| 9 |
+
|
| 10 |
+
## Execution
|
| 11 |
+
|
| 12 |
+
### Review repository contracts and implementation
|
| 13 |
+
|
| 14 |
+
- Read AGENTS.md, SKILLS.md, both mandatory contracts, public models/protocols, application
|
| 15 |
+
services, CLI composition, backend configurations, environment specs, and evidence
|
| 16 |
+
summaries.
|
| 17 |
+
- Compare documented commands and APIs with parser options and source signatures.
|
| 18 |
+
- Identify current, historical, and evidence-only documents.
|
| 19 |
+
|
| 20 |
+
### Draft the documentation architecture
|
| 21 |
+
|
| 22 |
+
- Store gap analysis and baseline evidence in `.agents/drafts/` and `.agents/work_data/`.
|
| 23 |
+
- Define README as the model-card and quick-start entry point.
|
| 24 |
+
- Define focused documents for architecture, configuration, evaluation, workflows,
|
| 25 |
+
testing/quality, and project status.
|
| 26 |
+
- Keep immutable historical reports unchanged except for navigational corrections.
|
| 27 |
+
|
| 28 |
+
### Refine public documentation
|
| 29 |
+
|
| 30 |
+
- Add `config.json` as a versioned machine-readable repository/model manifest.
|
| 31 |
+
- Expand Hugging Face-style metadata and tags without inventing unavailable model fields.
|
| 32 |
+
- Document all three alignment models/backends and every material configuration family.
|
| 33 |
+
- Document Python and CLI usage for annotation, benchmarking, and offline readiness.
|
| 34 |
+
- Define schema fields, timestamp semantics, failure policies, output artifacts, and
|
| 35 |
+
provenance requirements.
|
| 36 |
+
- Publish one performance comparison with explicit gold coverage and non-accuracy
|
| 37 |
+
caveats.
|
| 38 |
+
- Make current versus archival evidence and canonical artifact paths explicit.
|
| 39 |
+
|
| 40 |
+
### Verify acceptance gates
|
| 41 |
+
|
| 42 |
+
- Check local Markdown links and referenced repository paths.
|
| 43 |
+
- Run CLI help for every command.
|
| 44 |
+
- Run Ruff, mypy, import-linter, deterministic tests, architecture tests, header tests,
|
| 45 |
+
fresh coverage, and per-file coverage enforcement.
|
| 46 |
+
- Record observed commands/results in `.agents/logs/`.
|
| 47 |
+
- Review AGENTS.md and SKILLS.md at completion and update only if a durable gap is found.
|
| 48 |
+
|
| 49 |
+
## Acceptance criteria
|
| 50 |
+
|
| 51 |
+
- A new user can install an environment and run each documented entry point without
|
| 52 |
+
consulting stage reports.
|
| 53 |
+
- Every documented command has passed at least syntax/help-level verification.
|
| 54 |
+
- Configuration tables match typed config fields and defaults in source.
|
| 55 |
+
- `config.json` is valid JSON and its canonical values agree with `pyproject.toml`, the
|
| 56 |
+
engineering contract, and pinned backend configuration defaults.
|
| 57 |
+
- Metrics are tied to exact sample sizes and reference fields.
|
| 58 |
+
- Architecture documentation covers all current layers and modules.
|
| 59 |
+
- Tests remain at or above the repository gate: >=90% per implementation file and >=90%
|
| 60 |
+
aggregate coverage.
|
| 61 |
+
- Documentation-only work leaves source data/results unchanged.
|
| 62 |
+
- Work is committed on a branch and merged to main without a remote push.
|
| 63 |
+
- Final git status is clean.
|
|
@@ -0,0 +1,24 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Documentation Final Verification
|
| 2 |
+
|
| 3 |
+
Date: 2026-10-30
|
| 4 |
+
|
| 5 |
+
| Check | Result |
|
| 6 |
+
|---|---|
|
| 7 |
+
| Full deterministic tests | 635 passed, 3 skipped |
|
| 8 |
+
| Coverage run | 635 passed, 3 skipped |
|
| 9 |
+
| Aggregate implementation line coverage | 98.74% |
|
| 10 |
+
| Lowest implementation-file coverage | 93.33% (`backends/whisperx/worker.py`) |
|
| 11 |
+
| Ruff lint | Pass |
|
| 12 |
+
| Ruff format | 150 files already formatted |
|
| 13 |
+
| Strict mypy | Pass across 53 source files |
|
| 14 |
+
| Import Linter | 7 kept, 0 broken |
|
| 15 |
+
| Architecture tests | 10 passed |
|
| 16 |
+
| Header/manifest/link/per-file tests | Pass |
|
| 17 |
+
| CLI top-level and subcommand help | Pass |
|
| 18 |
+
| Conda/pip setup command help | Pass |
|
| 19 |
+
| Named `speech-text-data-aligners` environment | Python 3.12.15 available |
|
| 20 |
+
| Local Markdown links | All maintained targets resolve |
|
| 21 |
+
| Source metadata/results | Unchanged |
|
| 22 |
+
|
| 23 |
+
Coverage remains within the repository gate for every non-empty implementation file.
|
| 24 |
+
Heavyweight integration tests remain opt-in and were skipped in ordinary CI.
|
|
@@ -0,0 +1,72 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Documentation Review Baseline
|
| 2 |
+
|
| 3 |
+
Captured: 2026-10-30T15:27:45Z
|
| 4 |
+
|
| 5 |
+
## Repository state
|
| 6 |
+
|
| 7 |
+
- Base branch: `main`
|
| 8 |
+
- Review branch: `docs/documentation-refinement`
|
| 9 |
+
- Tracked implementation modules: 43 non-empty package Python files plus 10 empty
|
| 10 |
+
package markers
|
| 11 |
+
- Test modules mirror core, data, application, benchmarking, runtime, backend, CLI,
|
| 12 |
+
architecture, quality, and opt-in integration boundaries
|
| 13 |
+
|
| 14 |
+
## Observed deterministic quality results
|
| 15 |
+
|
| 16 |
+
| Gate | Result |
|
| 17 |
+
|---|---|
|
| 18 |
+
| Pytest | 633 passed, 3 skipped in 0.90 s |
|
| 19 |
+
| Coverage run | 633 passed, 3 skipped in 1.21 s |
|
| 20 |
+
| Aggregate line coverage | 98.74% |
|
| 21 |
+
| Lowest implementation-file coverage | 93.33% (`backends/whisperx/worker.py`) |
|
| 22 |
+
| Shared-layer minimum | 95.65% (`core/extraction.py`) |
|
| 23 |
+
| Ruff | All checks passed |
|
| 24 |
+
| mypy strict | Success: no issues found in 53 source files |
|
| 25 |
+
| Import Linter | 7 contracts kept, 0 broken |
|
| 26 |
+
| Architecture tests | 10 passed |
|
| 27 |
+
| Header/per-file quality tests | 4 passed |
|
| 28 |
+
|
| 29 |
+
## Current documentation inventory
|
| 30 |
+
|
| 31 |
+
Current guidance:
|
| 32 |
+
|
| 33 |
+
- `README.md`
|
| 34 |
+
- `docs/ARCHITECTURE.md`
|
| 35 |
+
- `docs/WORKFLOWS.md`
|
| 36 |
+
- `docs/PROJECT_STATUS.md`
|
| 37 |
+
- `docs/ALIGNER_ENGINEERING_CONTRACT.md`
|
| 38 |
+
- `docs/MODULAR_ARCHITECTURE_CONTRACT.md`
|
| 39 |
+
- `docs/W1_T_SEMANTIC_READINESS.md`
|
| 40 |
+
|
| 41 |
+
Decision and empirical evidence:
|
| 42 |
+
|
| 43 |
+
- `docs/FINAL_ALIGNER_SELECTION.md`
|
| 44 |
+
- `docs/POSTFINAL_THREE_ALIGNER_COMPARISON.md`
|
| 45 |
+
- `results/final/`
|
| 46 |
+
- `results/postfinal/`
|
| 47 |
+
- `results/w1t/`
|
| 48 |
+
|
| 49 |
+
Historical acceptance/provenance:
|
| 50 |
+
|
| 51 |
+
- `docs/STAGE_0_REPORT.md`
|
| 52 |
+
- `docs/STAGE_1_REPORT.md`
|
| 53 |
+
- `docs/STAGE_2_REPORT.md`
|
| 54 |
+
- `docs/STAGE_3_REPORT.md`
|
| 55 |
+
- `docs/STAGE_4_REPORT.md`
|
| 56 |
+
- `docs/STAGE_5_FINAL_ACCEPTANCE_REPORT.md`
|
| 57 |
+
- `docs/POST_STAGE5_DOCS_LICENSE_REPORT.md`
|
| 58 |
+
|
| 59 |
+
## Confirmed implementation facts for documentation
|
| 60 |
+
|
| 61 |
+
- Input columns are `line_id,audio_id,en_transcript,ru_translation`.
|
| 62 |
+
- Audio resolves strictly as `<audio_dir>/<audio_id>.wav`.
|
| 63 |
+
- Output appends six first/last lexical boundary fields and preserves source cells/order.
|
| 64 |
+
- CLI commands are `annotate`, `benchmark`, and `readiness`.
|
| 65 |
+
- Annotation accepts one backend and a backend-specific JSON object.
|
| 66 |
+
- Benchmark accepts repeated backend names and a JSON object keyed by backend name.
|
| 67 |
+
- Readiness accepts a provider/evidence catalog and writes a new output directory.
|
| 68 |
+
- Root public API exports annotation, benchmarking, neutral models, and `AlignmentBackend`.
|
| 69 |
+
- Specialized readiness/materialization services remain directly importable from their
|
| 70 |
+
application modules.
|
| 71 |
+
- Manual gold is limited to `first_word_text` and `first_word_end_s` on five rows.
|
| 72 |
+
- Pairwise timing differences are disagreement diagnostics, not accuracy estimates.
|
|
@@ -8,3 +8,10 @@ coverage.json
|
|
| 8 |
*.egg-info/
|
| 9 |
dist/
|
| 10 |
build/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 8 |
*.egg-info/
|
| 9 |
dist/
|
| 10 |
build/
|
| 11 |
+
.agents/config
|
| 12 |
+
.agents/user_prompts
|
| 13 |
+
.agents/user_scripts
|
| 14 |
+
.agents/logs/orchestrator-*.log
|
| 15 |
+
.*.swp
|
| 16 |
+
.*.swo
|
| 17 |
+
.agents/logs/codex-*.md
|
|
@@ -113,6 +113,13 @@ For each empirical run record at least:
|
|
| 113 |
|
| 114 |
Keep README and `docs/` aligned with actual commands and public Python APIs. Do not document commands that have not at least passed syntax/help-level checks, and clearly label commands that require unavailable hardware/data.
|
| 115 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 116 |
## Scope discipline
|
| 117 |
|
| 118 |
Do not refactor unrelated code. Do not change source transcripts or Russian translations. Do not overwrite source CSVs by default. Do not commit generated model weights, caches, large alignment outputs, WAV data, or credentials.
|
|
|
|
| 113 |
|
| 114 |
Keep README and `docs/` aligned with actual commands and public Python APIs. Do not document commands that have not at least passed syntax/help-level checks, and clearly label commands that require unavailable hardware/data.
|
| 115 |
|
| 116 |
+
Maintain `config.json` as a valid machine-readable project/model manifest, not as a
|
| 117 |
+
Transformers checkpoint configuration. Keep its schema, package version, registered
|
| 118 |
+
backends, model identities, data contract, evaluation scope, and canonical output hash
|
| 119 |
+
consistent with source and evidence through deterministic tests. Keep a documentation
|
| 120 |
+
map that distinguishes current guidance, contracts, empirical evidence, and historical
|
| 121 |
+
acceptance reports. Validate local Markdown links before completion.
|
| 122 |
+
|
| 123 |
## Scope discipline
|
| 124 |
|
| 125 |
Do not refactor unrelated code. Do not change source transcripts or Russian translations. Do not overwrite source CSVs by default. Do not commit generated model weights, caches, large alignment outputs, WAV data, or credentials.
|
|
@@ -0,0 +1,259 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Architecture
|
| 2 |
+
|
| 3 |
+
Speech-Text Data Aligners is a library-first forced-alignment framework. The core
|
| 4 |
+
contract accepts WAV audio plus a known, authoritative transcript and returns ordered
|
| 5 |
+
backend-neutral timed words. The CLI is a composition and presentation layer.
|
| 6 |
+
|
| 7 |
+
The mandatory design rules are in
|
| 8 |
+
[`docs/MODULAR_ARCHITECTURE_CONTRACT.md`](docs/MODULAR_ARCHITECTURE_CONTRACT.md).
|
| 9 |
+
Runtime alignment semantics are in
|
| 10 |
+
[`docs/ALIGNER_ENGINEERING_CONTRACT.md`](docs/ALIGNER_ENGINEERING_CONTRACT.md).
|
| 11 |
+
|
| 12 |
+
## Design Goals
|
| 13 |
+
|
| 14 |
+
- Keep authoritative reference text immutable.
|
| 15 |
+
- Keep core logic deterministic and independent of model/runtime packages.
|
| 16 |
+
- Make MFA, NeMo, and WhisperX interchangeable through `AlignmentBackend`.
|
| 17 |
+
- Contain native formats inside their adapters.
|
| 18 |
+
- Provide reusable Python services rather than CLI-only workflows.
|
| 19 |
+
- Keep optional runtimes independently unavailable.
|
| 20 |
+
- Preserve source metadata and deterministic output order.
|
| 21 |
+
- Represent failure explicitly and never fabricate timing.
|
| 22 |
+
|
| 23 |
+
## Layer and Dependency Direction
|
| 24 |
+
|
| 25 |
+
```text
|
| 26 |
+
CLI --------------------> Application --------------------> Core
|
| 27 |
+
| ^
|
| 28 |
+
+----------> Data -------------+
|
| 29 |
+
+----------> Benchmarking -----+
|
| 30 |
+
+----------> Runtime ----------+
|
| 31 |
+
|
| 32 |
+
CLI -- lazy composition --> Concrete Backends
|
| 33 |
+
| |
|
| 34 |
+
+------------> Core
|
| 35 |
+
+------------> Runtime
|
| 36 |
+
```
|
| 37 |
+
|
| 38 |
+
Enforced consequences:
|
| 39 |
+
|
| 40 |
+
- Core does not import application, data, benchmarking, runtime, CLI, or backends.
|
| 41 |
+
- Data and benchmarking do not import concrete backends or CLI.
|
| 42 |
+
- Application services receive `AlignmentBackend` implementations by injection and do
|
| 43 |
+
not branch on concrete backend names.
|
| 44 |
+
- Concrete backend packages may depend on core/runtime and their own native modules,
|
| 45 |
+
but never on a sibling backend.
|
| 46 |
+
- CLI composition may select a registry factory; it does not own alignment logic.
|
| 47 |
+
- Backend factories are lazy dotted references, so importing the public library does
|
| 48 |
+
not load heavyweight aligner dependencies.
|
| 49 |
+
|
| 50 |
+
## Current Package Tree
|
| 51 |
+
|
| 52 |
+
```text
|
| 53 |
+
src/speech_text_data_aligners/
|
| 54 |
+
|-- __init__.py stable public facade
|
| 55 |
+
|-- core/
|
| 56 |
+
| |-- models.py immutable neutral request/result/provenance values
|
| 57 |
+
| |-- protocols.py AlignmentBackend and structural contracts
|
| 58 |
+
| |-- errors.py project exception/failure taxonomy
|
| 59 |
+
| |-- normalization.py lexical plans and monotonic reconciliation
|
| 60 |
+
| |-- validation.py timing, lexical, and publication-path invariants
|
| 61 |
+
| |-- extraction.py first/last word output fields
|
| 62 |
+
| `-- readiness.py semantic-boundary mapping and consensus
|
| 63 |
+
|-- data/
|
| 64 |
+
| |-- metadata.py preserving, atomic CSV input/output
|
| 65 |
+
| |-- audio.py audio resolution and lightweight WAV probing
|
| 66 |
+
| `-- readiness.py frozen readiness cohort/hash verification
|
| 67 |
+
|-- application/
|
| 68 |
+
| |-- annotate.py reusable annotation orchestration
|
| 69 |
+
| |-- benchmark.py identical-request benchmark orchestration
|
| 70 |
+
| |-- readiness.py evidence reuse and offline readiness execution
|
| 71 |
+
| `-- materialize.py accepted-evidence full-word timing materialization
|
| 72 |
+
|-- benchmarking/
|
| 73 |
+
| |-- metrics.py sparse-gold, runtime, and pairwise metrics
|
| 74 |
+
| |-- reports.py deterministic JSON/CSV/Markdown reporting
|
| 75 |
+
| `-- readiness.py readiness distributions and sensitivity summaries
|
| 76 |
+
|-- runtime/
|
| 77 |
+
| |-- process.py shell-free injectable subprocess boundary
|
| 78 |
+
| |-- artifacts.py temporary/retained artifact lifecycle
|
| 79 |
+
| |-- receipts.py hashes, host facts, and run receipts
|
| 80 |
+
| `-- alignment_evidence.py full-word evidence integrity and verification
|
| 81 |
+
|-- backends/
|
| 82 |
+
| |-- registry.py immutable lazy backend factories
|
| 83 |
+
| |-- nemo/
|
| 84 |
+
| | |-- config.py typed NFA runtime/model configuration
|
| 85 |
+
| | |-- manifest.py reference/output manifest parsing
|
| 86 |
+
| | |-- runner.py isolated NFA execution
|
| 87 |
+
| | |-- ctm.py CTM parsing/mapping
|
| 88 |
+
| | `-- backend.py neutral NFA adapter
|
| 89 |
+
| |-- mfa/
|
| 90 |
+
| | |-- config.py typed MFA runtime/model configuration
|
| 91 |
+
| | |-- corpus.py temporary corpus/label construction
|
| 92 |
+
| | |-- runner.py isolated MFA execution
|
| 93 |
+
| | |-- textgrid.py TextGrid parsing/mapping
|
| 94 |
+
| | `-- backend.py neutral MFA adapter
|
| 95 |
+
| `-- whisperx/
|
| 96 |
+
| |-- config.py typed worker/runtime/model configuration
|
| 97 |
+
| |-- segments.py full-duration reference segments
|
| 98 |
+
| |-- provider.py isolated-worker transport
|
| 99 |
+
| |-- worker.py lazy heavyweight native runtime
|
| 100 |
+
| |-- parser.py native word parsing/mapping
|
| 101 |
+
| `-- backend.py neutral WhisperX adapter
|
| 102 |
+
`-- cli/
|
| 103 |
+
|-- main.py command dispatch and typed-error exit status
|
| 104 |
+
|-- annotate.py annotation argument adapter
|
| 105 |
+
|-- benchmark.py benchmark argument adapter
|
| 106 |
+
`-- readiness.py readiness catalog/composition adapter
|
| 107 |
+
```
|
| 108 |
+
|
| 109 |
+
Empty package-marker `__init__.py` files are deliberately zero bytes. Each non-empty
|
| 110 |
+
implementation module has a focused test module and meets its individual line-coverage
|
| 111 |
+
gate.
|
| 112 |
+
|
| 113 |
+
## Module Responsibilities
|
| 114 |
+
|
| 115 |
+
| Area | Owns | Does not own |
|
| 116 |
+
|---|---|---|
|
| 117 |
+
| `core` | Neutral values, normalization/reconciliation, validation, extraction, readiness rules | CSV, subprocesses, native formats, model loading |
|
| 118 |
+
| `data` | Metadata/audio/cohort input and source preservation | Backend selection or metrics |
|
| 119 |
+
| `application` | Reuse-case orchestration and publication policy | Native parsing or benchmark math |
|
| 120 |
+
| `benchmarking` | Deterministic metrics and report rendering | Backend execution or native result dictionaries |
|
| 121 |
+
| `runtime` | Process, artifact, receipt, and evidence integrity primitives | Application row loops or model semantics |
|
| 122 |
+
| `backends.*` | Config, native inputs/outputs, execution, parsing, mapping | Metadata orchestration or sibling backend imports |
|
| 123 |
+
| `cli` | Argument parsing, backend composition, presentation | Alignment, CSV loops, metric computation |
|
| 124 |
+
|
| 125 |
+
## Shared Alignment Contract
|
| 126 |
+
|
| 127 |
+
```python
|
| 128 |
+
result = backend.align(request)
|
| 129 |
+
outcomes = backend.align_batch(requests)
|
| 130 |
+
```
|
| 131 |
+
|
| 132 |
+
`AlignmentRequest` contains:
|
| 133 |
+
|
| 134 |
+
- caller/request identity;
|
| 135 |
+
- resolved WAV path;
|
| 136 |
+
- exact authoritative transcript;
|
| 137 |
+
- probed WAV duration;
|
| 138 |
+
- immutable JSON-safe context.
|
| 139 |
+
|
| 140 |
+
A successful `AlignmentResult` contains one `WordAlignment` per original lexical word,
|
| 141 |
+
plus provenance, diagnostics, and measured runtime. A failure is represented by
|
| 142 |
+
`AlignmentFailure`; it contains no timing fields. `AlignmentOutcome` enforces exactly
|
| 143 |
+
one result/failure variant.
|
| 144 |
+
|
| 145 |
+
The root package exports `AlignmentBackend`, neutral request/result/failure/provenance
|
| 146 |
+
models, annotation services, and benchmark services. Custom backends can implement the
|
| 147 |
+
protocol without importing a concrete adapter.
|
| 148 |
+
|
| 149 |
+
## Transcript and Alignment Flow
|
| 150 |
+
|
| 151 |
+
```text
|
| 152 |
+
source metadata
|
| 153 |
+
|
|
| 154 |
+
+-- immutable en_transcript
|
| 155 |
+
|
|
| 156 |
+
+-- resolve <audio_dir>/<audio_id>.wav and probe duration
|
| 157 |
+
v
|
| 158 |
+
lexical preflight and original-character mapping
|
| 159 |
+
v
|
| 160 |
+
backend-specific reversible reference representation
|
| 161 |
+
v
|
| 162 |
+
native model/process execution
|
| 163 |
+
v
|
| 164 |
+
backend-local native output parsing
|
| 165 |
+
v
|
| 166 |
+
shared monotonic reconciliation and validation
|
| 167 |
+
v
|
| 168 |
+
WordAlignment / AlignmentFailure
|
| 169 |
+
v
|
| 170 |
+
first/last extraction or benchmark observation
|
| 171 |
+
```
|
| 172 |
+
|
| 173 |
+
Backends may normalize punctuation, case, numbers, abbreviations, or native token
|
| 174 |
+
units internally. Every successful native token must map deterministically to the
|
| 175 |
+
original transcript's lexical words. Shared validation rejects incomplete coverage,
|
| 176 |
+
non-finite/negative/reversed/overlapping timing, and out-of-audio values. It never
|
| 177 |
+
clamps, interpolates, extrapolates, or invents a timestamp.
|
| 178 |
+
|
| 179 |
+
## Native Boundary Containment
|
| 180 |
+
|
| 181 |
+
| Backend | Native structures contained locally | External boundary |
|
| 182 |
+
|---|---|---|
|
| 183 |
+
| NeMo | Reference/output manifests and word CTM rows | Isolated NFA process |
|
| 184 |
+
| MFA | Temporary WAV/LAB corpus and TextGrid tiers | Isolated Conda MFA process |
|
| 185 |
+
| WhisperX | Reference segments and word-segment dictionaries | Isolated heavyweight worker |
|
| 186 |
+
|
| 187 |
+
Only neutral models, diagnostics, and provenance cross these boundaries. Parent-process
|
| 188 |
+
imports do not require NeMo, MFA, WhisperX, Torch, or Torchaudio.
|
| 189 |
+
|
| 190 |
+
## Annotation Publication
|
| 191 |
+
|
| 192 |
+
`application.annotate` prepares all source rows, invokes one injected backend, validates
|
| 193 |
+
and merges outcomes in source order, extracts six boundary columns, and atomically
|
| 194 |
+
publishes the enriched CSV. Continue-on-error preserves failed source rows with blank
|
| 195 |
+
alignment cells and structured diagnostics. Strict mode publishes no annotated output
|
| 196 |
+
after a row failure.
|
| 197 |
+
|
| 198 |
+
## Benchmark Composition
|
| 199 |
+
|
| 200 |
+
`application.benchmark` constructs one immutable ordered request tuple and supplies it
|
| 201 |
+
to every injected backend. It joins sparse manual gold by `(line_id, audio_id)`, scores
|
| 202 |
+
available first-word text/end fields, and compares successful first/last boundaries
|
| 203 |
+
pairwise. Reports are backend-neutral; native artifact paths appear only as opaque
|
| 204 |
+
references.
|
| 205 |
+
|
| 206 |
+
## Offline Readiness and Evidence
|
| 207 |
+
|
| 208 |
+
The readiness extension retains the same backend isolation:
|
| 209 |
+
|
| 210 |
+
```text
|
| 211 |
+
metadata + frozen boundaries + WAVs
|
| 212 |
+
|
|
| 213 |
+
v
|
| 214 |
+
cohort/hash verification
|
| 215 |
+
|
|
| 216 |
+
+-- verified full-word evidence capsules --> reuse audit
|
| 217 |
+
|
|
| 218 |
+
`-- missing/rejected case-backend pairs --> injected AlignmentBackend
|
| 219 |
+
|
|
| 220 |
+
v
|
| 221 |
+
full-word evidence verification
|
| 222 |
+
|
|
| 223 |
+
v
|
| 224 |
+
boundary mapping + consensus/reporting
|
| 225 |
+
```
|
| 226 |
+
|
| 227 |
+
`core.readiness` owns mapping/extraction/consensus, `data.readiness` owns cohort
|
| 228 |
+
verification, `runtime.alignment_evidence` owns capsule integrity, and
|
| 229 |
+
`application.readiness` orchestrates reuse/execution. `application.materialize`
|
| 230 |
+
materializes deterministic full-word timing from accepted immutable evidence without
|
| 231 |
+
importing or executing an aligner.
|
| 232 |
+
|
| 233 |
+
## Optional Dependency and Registry Behavior
|
| 234 |
+
|
| 235 |
+
`BackendRegistry` stores immutable dotted factories. Construction imports only the
|
| 236 |
+
selected backend module and validates the result against `AlignmentBackend`. An absent
|
| 237 |
+
optional runtime raises an actionable `BackendUnavailableError` only when selected.
|
| 238 |
+
One backend's absence does not affect backend-neutral imports or sibling construction.
|
| 239 |
+
|
| 240 |
+
## Architecture Verification
|
| 241 |
+
|
| 242 |
+
Release gates enforce:
|
| 243 |
+
|
| 244 |
+
- AST/import-boundary tests for prohibited directions and heavyweight imports;
|
| 245 |
+
- seven Import Linter contracts;
|
| 246 |
+
- backend sibling isolation;
|
| 247 |
+
- public-library tests with fake/injected backends;
|
| 248 |
+
- optional dependency absence;
|
| 249 |
+
- focused implementation-to-test mapping;
|
| 250 |
+
- per-file and aggregate coverage.
|
| 251 |
+
|
| 252 |
+
The current observed gate is documented in [`docs/TESTING.md`](docs/TESTING.md).
|
| 253 |
+
|
| 254 |
+
## Evolution Rules
|
| 255 |
+
|
| 256 |
+
Adding a backend should require a config, native input/execution/parser adapter, and
|
| 257 |
+
focused tests. It should not require changes to metadata I/O, common normalization,
|
| 258 |
+
first/last extraction, benchmark metrics, or the CLI beyond registry composition.
|
| 259 |
+
Shared helpers are introduced only when semantics are genuinely identical.
|
|
@@ -1,59 +1,132 @@
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
---
|
| 4 |
|
| 5 |
# Speech-Text Data Aligners
|
| 6 |
|
| 7 |
-
|
| 8 |
-
|
| 9 |
-
|
|
|
|
| 10 |
|
| 11 |
-
|
|
|
|
|
|
|
| 12 |
|
| 13 |
-
|
| 14 |
-
|
| 15 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 16 |
- Source-preserving, deterministic CSV annotation with atomic publication.
|
| 17 |
-
-
|
| 18 |
-
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 19 |
|
| 20 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 21 |
|
| 22 |
-
|
| 23 |
-
|
| 24 |
-
| NVIDIA NeMo Forced Aligner (NFA) | Accepted | Real NVIDIA A10G smoke, five-row benchmark, and strict 36-row candidate passed |
|
| 25 |
-
| Montreal Forced Aligner (MFA) 3.x | Accepted | **Selected final backend**; five-row and strict full-36 validation passed |
|
| 26 |
-
| WhisperX 3.8.6 | Accepted | Five-row and strict full-36 validation passed; accepted alternative |
|
| 27 |
|
| 28 |
-
|
| 29 |
-
|
|
|
|
|
|
|
|
|
|
| 30 |
|
| 31 |
-
|
|
|
|
|
|
|
|
|
|
| 32 |
|
| 33 |
-
|
|
|
|
|
|
|
| 34 |
|
| 35 |
```text
|
| 36 |
line_id,audio_id,en_transcript,ru_translation
|
| 37 |
```
|
| 38 |
|
| 39 |
-
Audio
|
| 40 |
-
|
| 41 |
|
| 42 |
```text
|
| 43 |
-
first_word_text,first_word_start_s,first_word_end_s,
|
| 44 |
-
last_word_text,last_word_start_s,last_word_end_s
|
| 45 |
```
|
| 46 |
|
| 47 |
-
|
| 48 |
-
|
| 49 |
|
| 50 |
-
|
|
|
|
|
|
|
| 51 |
|
| 52 |
-
|
| 53 |
-
|
| 54 |
-
|
| 55 |
|
| 56 |
-
|
|
|
|
|
|
|
| 57 |
|
| 58 |
```python
|
| 59 |
from pathlib import Path
|
|
@@ -69,61 +142,146 @@ run = annotate_metadata(
|
|
| 69 |
backend=MfaBackend(MfaConfig(environment_name="speech-aligners-mfa")),
|
| 70 |
options=AnnotationOptions(strict=False),
|
| 71 |
)
|
|
|
|
| 72 |
print(run.receipt.success_count, run.receipt.failure_count)
|
| 73 |
```
|
| 74 |
|
| 75 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 76 |
|
| 77 |
```bash
|
| 78 |
-
|
| 79 |
metadata.csv wavs annotated.csv \
|
| 80 |
--backend mfa \
|
| 81 |
-
--backend-config-json '{"environment_name":"speech-aligners-mfa"}'
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 82 |
```
|
| 83 |
|
| 84 |
-
|
| 85 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 86 |
|
| 87 |
## Documentation
|
| 88 |
|
| 89 |
-
|
| 90 |
-
- [Architecture](docs/ARCHITECTURE.md)
|
| 91 |
-
- [Workflows](docs/WORKFLOWS.md)
|
| 92 |
-
- [Aligner engineering contract](docs/ALIGNER_ENGINEERING_CONTRACT.md)
|
| 93 |
-
- [Modular architecture contract](docs/MODULAR_ARCHITECTURE_CONTRACT.md)
|
| 94 |
-
- [Stage 5 final acceptance report](docs/STAGE_5_FINAL_ACCEPTANCE_REPORT.md)
|
| 95 |
-
- [Post-final three-aligner comparison](docs/POSTFINAL_THREE_ALIGNER_COMPARISON.md)
|
| 96 |
-
- [Final aligner selection](docs/FINAL_ALIGNER_SELECTION.md)
|
| 97 |
-
- [W1-T offline semantic-readiness timing](docs/W1_T_SEMANTIC_READINESS.md)
|
| 98 |
|
| 99 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 100 |
|
| 101 |
-
|
| 102 |
-
MFA extension has closed the full-corpus comparison. All three backends now have successful
|
| 103 |
-
36/36 candidates. On the five shared manual-gold rows, first-word-end MAE is 0.0162 s for
|
| 104 |
-
MFA, 0.0154 s for WhisperX, and 0.2210 s for NeMo.
|
| 105 |
|
| 106 |
-
|
| 107 |
-
|
| 108 |
-
|
| 109 |
-
|
|
|
|
| 110 |
|
| 111 |
-
|
| 112 |
-
|
| 113 |
-
but no manual ground-truth validation. Pairwise agreement remains review evidence, not
|
| 114 |
-
ground truth.
|
| 115 |
|
| 116 |
-
## License
|
| 117 |
|
| 118 |
-
Licensed under the [Apache License 2.0](LICENSE).
|
| 119 |
-
[official license text](https://www.apache.org/licenses/LICENSE-2.0) is also available
|
| 120 |
-
from the Apache Software Foundation.
|
| 121 |
|
| 122 |
Copyright (c) 2026 VertoX-AI. All rights reserved.
|
| 123 |
|
| 124 |
## Author
|
| 125 |
|
| 126 |
-
Patrick Lumbantobing
|
| 127 |
|
| 128 |
## Citation
|
| 129 |
|
|
@@ -137,13 +295,13 @@ Patrick Lumbantobing — [VertoX-AI](https://huggingface.co/vertox-ai)
|
|
| 137 |
}
|
| 138 |
```
|
| 139 |
|
| 140 |
-
##
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 141 |
|
| 142 |
-
This project builds on [NVIDIA NeMo and NeMo Forced Aligner](https://docs.nvidia.com/nemo/speech/nightly/tools/nemo_forced_aligner.html),
|
| 143 |
-
[Montreal Forced Aligner](https://montreal-forced-aligner.readthedocs.io/en/latest/),
|
| 144 |
-
[WhisperX](https://github.com/m-bain/whisperX),
|
| 145 |
-
[PyTorch](https://pytorch.org/),
|
| 146 |
-
[Torchaudio](https://docs.pytorch.org/audio/stable/index.html), and
|
| 147 |
-
[Hugging Face](https://huggingface.co/). WhisperX uses Torchaudio's selected
|
| 148 |
-
[WAV2VEC2_ASR_BASE_960H bundle](https://docs.pytorch.org/audio/2.8.0/generated/torchaudio.pipelines.WAV2VEC2_ASR_BASE_960H.html).
|
| 149 |
Acknowledgment does not imply endorsement by any upstream project or organization.
|
|
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
| 3 |
+
language: en
|
| 4 |
+
tags:
|
| 5 |
+
- audio
|
| 6 |
+
- speech
|
| 7 |
+
- forced-alignment
|
| 8 |
+
- reference-text-alignment
|
| 9 |
+
- speech-processing
|
| 10 |
+
- mfa
|
| 11 |
+
- nemo
|
| 12 |
+
- whisperx
|
| 13 |
+
- python
|
| 14 |
+
- metric:mae
|
| 15 |
+
model-index:
|
| 16 |
+
- name: speech-text-data-aligners
|
| 17 |
+
results:
|
| 18 |
+
- task:
|
| 19 |
+
type: forced-alignment
|
| 20 |
+
name: Forced alignment
|
| 21 |
+
dataset:
|
| 22 |
+
type: local
|
| 23 |
+
name: five-row-first-word-manual-gold
|
| 24 |
+
metrics:
|
| 25 |
+
- type: mae
|
| 26 |
+
value: 0.0162
|
| 27 |
+
name: MFA first-word-end MAE in seconds
|
| 28 |
+
- type: mae
|
| 29 |
+
value: 0.0154
|
| 30 |
+
name: WhisperX first-word-end MAE in seconds
|
| 31 |
+
- type: mae
|
| 32 |
+
value: 0.2210
|
| 33 |
+
name: NeMo first-word-end MAE in seconds
|
| 34 |
---
|
| 35 |
|
| 36 |
# Speech-Text Data Aligners
|
| 37 |
|
| 38 |
+
Speech-Text Data Aligners (STA) is a library-first framework for forced alignment of
|
| 39 |
+
WAV audio against a known, authoritative English transcript. It estimates when the
|
| 40 |
+
supplied words occur in audio. It is not ASR and never replaces `en_transcript` with a
|
| 41 |
+
recognized transcript.
|
| 42 |
|
| 43 |
+
The repository provides one backend-neutral Python contract for Montreal Forced
|
| 44 |
+
Aligner (MFA), NVIDIA NeMo Forced Aligner (NFA), and WhisperX, plus deterministic
|
| 45 |
+
annotation, benchmarking, and offline semantic-readiness services.
|
| 46 |
|
| 47 |
+
This repository is a software framework and comparative evaluation package, not a
|
| 48 |
+
single neural checkpoint. [`config.json`](config.json) is its machine-readable
|
| 49 |
+
project/model manifest; it is not a Transformers `AutoConfig` file.
|
| 50 |
+
|
| 51 |
+
## Highlights
|
| 52 |
+
|
| 53 |
+
- Authoritative reference-text alignment with no silent ASR substitution.
|
| 54 |
+
- Reversible lexical mapping back to exact source transcript words.
|
| 55 |
+
- Strict finite, non-negative, monotonic, non-overlapping, WAV-bounded timestamps.
|
| 56 |
+
- No clamping, interpolation, extrapolation, or fabricated success/timing.
|
| 57 |
- Source-preserving, deterministic CSV annotation with atomic publication.
|
| 58 |
+
- Python-callable services; the CLI delegates to those services.
|
| 59 |
+
- Typed backend configuration, provenance, diagnostics, and optional native artifacts.
|
| 60 |
+
- Lazy backend isolation: missing NeMo/MFA/WhisperX dependencies do not break imports.
|
| 61 |
+
- Deterministic unit, architecture, header, manifest, and per-file coverage gates.
|
| 62 |
+
|
| 63 |
+
## Model and Configuration Matrix
|
| 64 |
+
|
| 65 |
+
| Backend | Pinned alignment model | Runtime baseline | Default device | Current role |
|
| 66 |
+
|---|---|---|---|---|
|
| 67 |
+
| MFA | `MontrealCorpusTools/english_mfa@1009e3c6c689f10875146a5e4e5c2233f8eaf922` (`english_us_mfa`, model 3.3.0) | MFA 3.4.2 | CPU | **Selected operational backend** |
|
| 68 |
+
| WhisperX | `WAV2VEC2_ASR_BASE_960H` / Torchaudio 2.8.0 bundle | WhisperX 3.8.6 | CPU | Accepted alternative |
|
| 69 |
+
| NeMo / NFA | `stt_en_fastconformer_hybrid_large_pc` | NeMo 3.0.0 / source `fd6a877539710e2b98f28c43272ff81312f83417` | CUDA | Accepted; not selected operationally |
|
| 70 |
+
|
| 71 |
+
All supported configuration fields, defaults, invariants, and JSON examples are in
|
| 72 |
+
[Configuration](docs/CONFIGURATION.md). Compact model, schema, and evaluation metadata
|
| 73 |
+
are in [`config.json`](config.json).
|
| 74 |
+
|
| 75 |
+
## Installation
|
| 76 |
+
|
| 77 |
+
Use Python 3.12 or newer. Keep each heavyweight backend in its dedicated environment.
|
| 78 |
+
The repository contains a deterministic development environment and separate backend
|
| 79 |
+
environment specifications:
|
| 80 |
|
| 81 |
+
```bash
|
| 82 |
+
conda env create -f environments/core-dev.yml
|
| 83 |
+
conda activate speech-aligners-core-dev
|
| 84 |
+
python -m pip install -e .
|
| 85 |
+
```
|
| 86 |
|
| 87 |
+
An existing environment named `speech-text-data-aligners` is also suitable when it
|
| 88 |
+
contains the packages pinned in [`requirements.txt`](requirements.txt):
|
|
|
|
|
|
|
|
|
|
| 89 |
|
| 90 |
+
```bash
|
| 91 |
+
conda deactivate
|
| 92 |
+
conda activate speech-text-data-aligners
|
| 93 |
+
python -m pip install -e . --no-deps
|
| 94 |
+
```
|
| 95 |
|
| 96 |
+
Backend environments are created separately from
|
| 97 |
+
[`environments/mfa.yml`](environments/mfa.yml), [`environments/nemo.yml`](environments/nemo.yml),
|
| 98 |
+
and [`environments/whisperx.yml`](environments/whisperx.yml). The parent CLI process
|
| 99 |
+
invokes the selected isolated runtime; see [Configuration](docs/CONFIGURATION.md).
|
| 100 |
|
| 101 |
+
## Data Contract
|
| 102 |
+
|
| 103 |
+
Input metadata must contain these columns:
|
| 104 |
|
| 105 |
```text
|
| 106 |
line_id,audio_id,en_transcript,ru_translation
|
| 107 |
```
|
| 108 |
|
| 109 |
+
Audio must be available at `<audio_dir>/<audio_id>.wav`. Annotation preserves every
|
| 110 |
+
source header, cell value, CSV row, and row position, then appends exactly:
|
| 111 |
|
| 112 |
```text
|
| 113 |
+
first_word_text,first_word_start_s,first_word_end_s,last_word_text,last_word_start_s,last_word_end_s
|
|
|
|
| 114 |
```
|
| 115 |
|
| 116 |
+
For a one-word utterance, first and last fields come from the same aligned word.
|
| 117 |
+
Successful timing must satisfy:
|
| 118 |
|
| 119 |
+
```text
|
| 120 |
+
0 <= first_start <= first_end <= last_start <= last_end <= audio_duration
|
| 121 |
+
```
|
| 122 |
|
| 123 |
+
Equality is allowed at backend resolution limits. NaN, infinity, negative values,
|
| 124 |
+
reversed/overlapping intervals, incomplete lexical coverage, and out-of-audio values
|
| 125 |
+
fail validation.
|
| 126 |
|
| 127 |
+
## Python Usage
|
| 128 |
+
|
| 129 |
+
The root package exposes the stable backend-neutral API:
|
| 130 |
|
| 131 |
```python
|
| 132 |
from pathlib import Path
|
|
|
|
| 142 |
backend=MfaBackend(MfaConfig(environment_name="speech-aligners-mfa")),
|
| 143 |
options=AnnotationOptions(strict=False),
|
| 144 |
)
|
| 145 |
+
|
| 146 |
print(run.receipt.success_count, run.receipt.failure_count)
|
| 147 |
```
|
| 148 |
|
| 149 |
+
Custom or test backends can implement `AlignmentBackend`; application code receives
|
| 150 |
+
them by dependency injection and does not branch on backend names.
|
| 151 |
+
|
| 152 |
+
For one backend:
|
| 153 |
+
|
| 154 |
+
```python
|
| 155 |
+
from pathlib import Path
|
| 156 |
+
|
| 157 |
+
from speech_text_data_aligners import benchmark_backend
|
| 158 |
+
from speech_text_data_aligners.backends.mfa.backend import MfaBackend
|
| 159 |
+
from speech_text_data_aligners.backends.mfa.config import MfaConfig
|
| 160 |
+
|
| 161 |
+
report = benchmark_backend(
|
| 162 |
+
backend=MfaBackend(MfaConfig()),
|
| 163 |
+
metadata_csv=Path("metadata.csv"),
|
| 164 |
+
reference_csv=Path("first-word-gold.csv"),
|
| 165 |
+
audio_dir=Path("wavs"),
|
| 166 |
+
)
|
| 167 |
+
```
|
| 168 |
+
|
| 169 |
+
Use `benchmark_backends` for an insertion-ordered mapping of multiple backend
|
| 170 |
+
instances. Specialized offline readiness services are documented in
|
| 171 |
+
[W1-T semantic readiness](docs/W1_T_SEMANTIC_READINESS.md).
|
| 172 |
+
|
| 173 |
+
## CLI Usage
|
| 174 |
+
|
| 175 |
+
After installation, `speech-text-aligners` and
|
| 176 |
+
`python -m speech_text_data_aligners.cli.main` are equivalent entry points.
|
| 177 |
+
|
| 178 |
+
Annotate with the selected MFA environment:
|
| 179 |
|
| 180 |
```bash
|
| 181 |
+
speech-text-aligners annotate \
|
| 182 |
metadata.csv wavs annotated.csv \
|
| 183 |
--backend mfa \
|
| 184 |
+
--backend-config-json '{"environment_name":"speech-aligners-mfa"}' \
|
| 185 |
+
--receipt annotated.receipt.json
|
| 186 |
+
```
|
| 187 |
+
|
| 188 |
+
Benchmark multiple backends. Benchmark configuration is one JSON object keyed by the
|
| 189 |
+
values passed to `--backend`:
|
| 190 |
+
|
| 191 |
+
```bash
|
| 192 |
+
speech-text-aligners benchmark \
|
| 193 |
+
metadata.csv first-word-gold.csv wavs \
|
| 194 |
+
--backend mfa \
|
| 195 |
+
--backend whisperx \
|
| 196 |
+
--backend-config-json '{"mfa":{"environment_name":"speech-aligners-mfa"},"whisperx":{"python_executable":"/path/to/whisperx/bin/python","device":"cpu"}}' \
|
| 197 |
+
--output-base reports/comparison
|
| 198 |
+
```
|
| 199 |
+
|
| 200 |
+
Run offline semantic readiness with an explicit provider/evidence catalog:
|
| 201 |
+
|
| 202 |
+
```bash
|
| 203 |
+
speech-text-aligners readiness \
|
| 204 |
+
source_metadata_en_only.csv boundaries.jsonl wavs results/w1t/new-attempt \
|
| 205 |
+
--catalog /absolute/path/to/catalog.json
|
| 206 |
```
|
| 207 |
|
| 208 |
+
Run `speech-text-aligners <command> --help` for the authoritative argument list.
|
| 209 |
+
Detailed workflows and output artifacts are in [Workflows](docs/WORKFLOWS.md).
|
| 210 |
+
|
| 211 |
+
## Failure and Publication Policy
|
| 212 |
+
|
| 213 |
+
- `--strict` aborts on the first row/backend failure and publishes no annotated CSV.
|
| 214 |
+
- Without `--strict`, failed rows retain their source cells, leave all six alignment
|
| 215 |
+
cells blank, and are recorded in a deterministic diagnostics JSON file.
|
| 216 |
+
- Existing output artifacts require `--overwrite`.
|
| 217 |
+
- Annotation writes a complete temporary CSV, flushes and fsyncs it, then atomically
|
| 218 |
+
publishes it. Input/output aliases are rejected before backend execution.
|
| 219 |
+
- Benchmarks retain structured failure observations and score only available
|
| 220 |
+
references; failures never become fabricated timestamps.
|
| 221 |
+
|
| 222 |
+
## Evaluation Summary
|
| 223 |
+
|
| 224 |
+
Manual gold exists for only `first_word_text` and `first_word_end_s` on five rows.
|
| 225 |
+
|
| 226 |
+
| Backend | Success | Exact first-word text | First-word-end MAE | Median | Maximum |
|
| 227 |
+
|---|---:|---:|---:|---:|---:|
|
| 228 |
+
| MFA | 5/5 | 5/5 | **0.0162 s** | **0.0110 s** | **0.0300 s** |
|
| 229 |
+
| WhisperX | 5/5 | 5/5 | **0.0154 s** | **0.0140 s** | **0.0310 s** |
|
| 230 |
+
| NeMo | 5/5 | 5/5 | **0.2210 s** | **0.0100 s** | **0.9200 s** |
|
| 231 |
+
|
| 232 |
+
MFA and WhisperX are effectively tied on this sparse gold; the 0.0008 s MAE difference
|
| 233 |
+
is not decision-grade. All three backends produced 36/36 structurally valid candidates.
|
| 234 |
+
At a 0.250 s material-disagreement threshold, MFA/WhisperX differs on 7 rows, MFA/NeMo
|
| 235 |
+
on 28, and WhisperX/NeMo on 32.
|
| 236 |
+
|
| 237 |
+
Only first-word text and first-word end have manual ground truth. First-word start and
|
| 238 |
+
both last-word boundaries have structural validation and pairwise comparison, but no
|
| 239 |
+
manual accuracy evidence. Pairwise agreement is diagnostic, not ground truth. Runtime
|
| 240 |
+
observations used different hosts, devices, and cache states and are secondary.
|
| 241 |
+
|
| 242 |
+
See [Evaluation](docs/EVALUATION.md) for metric definitions, model/configuration
|
| 243 |
+
coverage, full pairwise tables, W1-T evidence, caveats, and artifact paths.
|
| 244 |
+
|
| 245 |
+
## Engineering Status
|
| 246 |
+
|
| 247 |
+
The current checkout passes 635 deterministic tests with 3 heavyweight integration
|
| 248 |
+
tests skipped, 98.74% aggregate line coverage, all individual implementation-file
|
| 249 |
+
coverage gates, Ruff, strict mypy, and all 7 import-linter contracts. The current
|
| 250 |
+
lowest implementation file is 93.33%. Commands and interpretation are in
|
| 251 |
+
[Testing and Quality](docs/TESTING.md).
|
| 252 |
|
| 253 |
## Documentation
|
| 254 |
|
| 255 |
+
Current guidance:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 256 |
|
| 257 |
+
- [Architecture](ARCHITECTURE.md)
|
| 258 |
+
- [Configuration](docs/CONFIGURATION.md)
|
| 259 |
+
- [Evaluation](docs/EVALUATION.md)
|
| 260 |
+
- [Workflows](docs/WORKFLOWS.md)
|
| 261 |
+
- [Testing and Quality](docs/TESTING.md)
|
| 262 |
+
- [Project Status](docs/PROJECT_STATUS.md)
|
| 263 |
+
- [Documentation Map](docs/DOCUMENTATION.md)
|
| 264 |
|
| 265 |
+
Contracts and evidence:
|
|
|
|
|
|
|
|
|
|
| 266 |
|
| 267 |
+
- [Aligner Engineering Contract](docs/ALIGNER_ENGINEERING_CONTRACT.md)
|
| 268 |
+
- [Modular Architecture Contract](docs/MODULAR_ARCHITECTURE_CONTRACT.md)
|
| 269 |
+
- [Final Aligner Selection](docs/FINAL_ALIGNER_SELECTION.md)
|
| 270 |
+
- [Post-final Three-Aligner Comparison](docs/POSTFINAL_THREE_ALIGNER_COMPARISON.md)
|
| 271 |
+
- [W1-T Semantic Readiness](docs/W1_T_SEMANTIC_READINESS.md)
|
| 272 |
|
| 273 |
+
Historical stage reports remain unchanged as implementation and acceptance provenance;
|
| 274 |
+
they are not the primary operating guide.
|
|
|
|
|
|
|
| 275 |
|
| 276 |
+
## License
|
| 277 |
|
| 278 |
+
Licensed under the [Apache License 2.0](LICENSE).
|
|
|
|
|
|
|
| 279 |
|
| 280 |
Copyright (c) 2026 VertoX-AI. All rights reserved.
|
| 281 |
|
| 282 |
## Author
|
| 283 |
|
| 284 |
+
Patrick Lumbantobing - [VertoX-AI](https://huggingface.co/vertox-ai)
|
| 285 |
|
| 286 |
## Citation
|
| 287 |
|
|
|
|
| 295 |
}
|
| 296 |
```
|
| 297 |
|
| 298 |
+
## References
|
| 299 |
+
|
| 300 |
+
- [Hugging Face model cards](https://huggingface.co/docs/hub/en/model-cards)
|
| 301 |
+
- [NVIDIA NeMo Forced Aligner](https://docs.nvidia.com/nemo/speech/nightly/tools/nemo_forced_aligner.html)
|
| 302 |
+
- [Montreal Forced Aligner](https://montreal-forced-aligner.readthedocs.io/en/latest/)
|
| 303 |
+
- [WhisperX](https://github.com/m-bain/whisperX)
|
| 304 |
+
- [PyTorch](https://pytorch.org/)
|
| 305 |
+
- [Torchaudio WAV2VEC2_ASR_BASE_960H](https://docs.pytorch.org/audio/2.8.0/generated/torchaudio.pipelines.WAV2VEC2_ASR_BASE_960H.html)
|
| 306 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 307 |
Acknowledgment does not imply endorsement by any upstream project or organization.
|
|
@@ -83,3 +83,19 @@ Run all executable backends on the exact same input subset. Compare at least:
|
|
| 83 |
- runtime per utterance and total runtime where practical.
|
| 84 |
|
| 85 |
Do not infer accuracy of unannotated fields from backend agreement alone.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 83 |
- runtime per utterance and total runtime where practical.
|
| 84 |
|
| 85 |
Do not infer accuracy of unannotated fields from backend agreement alone.
|
| 86 |
+
|
| 87 |
+
## Skill: documentation and manifest integrity
|
| 88 |
+
|
| 89 |
+
Treat documentation as a tested interface. Keep the Hugging Face-style README,
|
| 90 |
+
root `ARCHITECTURE.md`, focused usage/config/evaluation/testing guides, and
|
| 91 |
+
`config.json` aligned with actual source behavior and retained evidence.
|
| 92 |
+
|
| 93 |
+
Use `config.json` as a versioned project/model manifest, not as a Transformers
|
| 94 |
+
checkpoint configuration or a direct CLI backend-config payload. Add deterministic
|
| 95 |
+
tests that bind its package version, schema fields, registered backends, model
|
| 96 |
+
identities, and canonical result hashes to source/evidence values.
|
| 97 |
+
|
| 98 |
+
Separate current operating guidance from contracts, empirical evidence, and historical
|
| 99 |
+
acceptance reports. Link all local documents from a maintained documentation map,
|
| 100 |
+
validate local Markdown links, and run every documented syntax/help/quality command.
|
| 101 |
+
Never turn pairwise agreement or structural success into an accuracy claim.
|
|
@@ -0,0 +1,162 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"schema_version": "speech-text-data-aligners/config.v1",
|
| 3 |
+
"project": {
|
| 4 |
+
"name": "speech-text-data-aligners",
|
| 5 |
+
"version": "0.1.0",
|
| 6 |
+
"package": "speech_text_data_aligners",
|
| 7 |
+
"python": ">=3.12",
|
| 8 |
+
"entrypoint": "speech-text-aligners",
|
| 9 |
+
"license": "Apache-2.0",
|
| 10 |
+
"task": "forced-alignment",
|
| 11 |
+
"languages": [
|
| 12 |
+
"en"
|
| 13 |
+
],
|
| 14 |
+
"author": "Patrick Lumbantobing, VertoX-AI",
|
| 15 |
+
"repository": "https://huggingface.co/vertox-ai/speech-text-data-aligners"
|
| 16 |
+
},
|
| 17 |
+
"data_contract": {
|
| 18 |
+
"transcript_column": "en_transcript",
|
| 19 |
+
"source_columns": [
|
| 20 |
+
"line_id",
|
| 21 |
+
"audio_id",
|
| 22 |
+
"en_transcript",
|
| 23 |
+
"ru_translation"
|
| 24 |
+
],
|
| 25 |
+
"audio_path_template": "<audio_dir>/<audio_id>.wav",
|
| 26 |
+
"alignment_columns": [
|
| 27 |
+
"first_word_text",
|
| 28 |
+
"first_word_start_s",
|
| 29 |
+
"first_word_end_s",
|
| 30 |
+
"last_word_text",
|
| 31 |
+
"last_word_start_s",
|
| 32 |
+
"last_word_end_s"
|
| 33 |
+
],
|
| 34 |
+
"timestamp_unit": "seconds",
|
| 35 |
+
"authoritative_transcript": true
|
| 36 |
+
},
|
| 37 |
+
"backend_registry": {
|
| 38 |
+
"operational_selection": "mfa",
|
| 39 |
+
"models": {
|
| 40 |
+
"mfa": {
|
| 41 |
+
"selected": true,
|
| 42 |
+
"config_class": "speech_text_data_aligners.backends.mfa.config.MfaConfig",
|
| 43 |
+
"runtime": {
|
| 44 |
+
"package": "montreal-forced-aligner",
|
| 45 |
+
"version": "3.4.2",
|
| 46 |
+
"source_revision": "d2dc283bd79667e086b1f93050a1855349f63f1f",
|
| 47 |
+
"kaldi_version": "5.5.1172",
|
| 48 |
+
"kalpy_version": "0.10.5",
|
| 49 |
+
"openfst_version": "1.8.4"
|
| 50 |
+
},
|
| 51 |
+
"model": {
|
| 52 |
+
"id": "MontrealCorpusTools/english_mfa",
|
| 53 |
+
"revision": "1009e3c6c689f10875146a5e4e5c2233f8eaf922",
|
| 54 |
+
"version": "3.3.0",
|
| 55 |
+
"dialect": "english_us_mfa"
|
| 56 |
+
},
|
| 57 |
+
"observed_evidence": {
|
| 58 |
+
"model_tree_sha256": "486bf1be991b0590f3554040a16502d033237f63d5e426ca231b92b0a9829ada"
|
| 59 |
+
}
|
| 60 |
+
},
|
| 61 |
+
"whisperx": {
|
| 62 |
+
"selected": false,
|
| 63 |
+
"config_class": "speech_text_data_aligners.backends.whisperx.config.WhisperXConfig",
|
| 64 |
+
"runtime": {
|
| 65 |
+
"package": "whisperx",
|
| 66 |
+
"version": "3.8.6",
|
| 67 |
+
"source_revision": "3ccc17b8de34f305300f8a3fd3c9f76ba820c0d0",
|
| 68 |
+
"torch_version": "2.8.0",
|
| 69 |
+
"torchaudio_version": "2.8.0"
|
| 70 |
+
},
|
| 71 |
+
"model": {
|
| 72 |
+
"id": "WAV2VEC2_ASR_BASE_960H",
|
| 73 |
+
"revision": "torchaudio-2.8.0",
|
| 74 |
+
"language": "en"
|
| 75 |
+
},
|
| 76 |
+
"observed_evidence": {
|
| 77 |
+
"model_asset_sha256": "488fd4f16de84438ffc945334278c1b9fb9b7159a806c1080b16111a958c945d"
|
| 78 |
+
}
|
| 79 |
+
},
|
| 80 |
+
"nemo": {
|
| 81 |
+
"selected": false,
|
| 82 |
+
"config_class": "speech_text_data_aligners.backends.nemo.config.NemoConfig",
|
| 83 |
+
"runtime": {
|
| 84 |
+
"package": "nemo-toolkit",
|
| 85 |
+
"version": "3.0.0",
|
| 86 |
+
"source_revision": "fd6a877539710e2b98f28c43272ff81312f83417",
|
| 87 |
+
"align_script_sha256": "1cfaf9526b47037204b7874c4432e4d5d07364a7fca1b840052c370d46ee2f03"
|
| 88 |
+
},
|
| 89 |
+
"model": {
|
| 90 |
+
"id": "stt_en_fastconformer_hybrid_large_pc",
|
| 91 |
+
"revision": null,
|
| 92 |
+
"resolution": "pretrained_name_or_local_checkpoint",
|
| 93 |
+
"local_checkpoint_preferred": true
|
| 94 |
+
},
|
| 95 |
+
"observed_evidence": {
|
| 96 |
+
"model_sha256": "37afe6627816825af7646c02f9ada5dbf58fe870e7a9cc60fffbd8dd412c8b61"
|
| 97 |
+
}
|
| 98 |
+
}
|
| 99 |
+
}
|
| 100 |
+
},
|
| 101 |
+
"evaluation": {
|
| 102 |
+
"manual_gold": {
|
| 103 |
+
"rows": 5,
|
| 104 |
+
"fields": [
|
| 105 |
+
"first_word_text",
|
| 106 |
+
"first_word_end_s"
|
| 107 |
+
],
|
| 108 |
+
"unvalidated_fields": [
|
| 109 |
+
"first_word_start_s",
|
| 110 |
+
"last_word_start_s",
|
| 111 |
+
"last_word_end_s"
|
| 112 |
+
]
|
| 113 |
+
},
|
| 114 |
+
"results": [
|
| 115 |
+
{
|
| 116 |
+
"backend": "mfa",
|
| 117 |
+
"success": "5/5",
|
| 118 |
+
"exact_first_word_text": "5/5",
|
| 119 |
+
"first_word_end_mae_s": 0.0162,
|
| 120 |
+
"first_word_end_median_s": 0.0110,
|
| 121 |
+
"first_word_end_max_s": 0.0300,
|
| 122 |
+
"observed_runtime_s": 274.9569
|
| 123 |
+
},
|
| 124 |
+
{
|
| 125 |
+
"backend": "whisperx",
|
| 126 |
+
"success": "5/5",
|
| 127 |
+
"exact_first_word_text": "5/5",
|
| 128 |
+
"first_word_end_mae_s": 0.0154,
|
| 129 |
+
"first_word_end_median_s": 0.0140,
|
| 130 |
+
"first_word_end_max_s": 0.0310,
|
| 131 |
+
"observed_runtime_s": 28.5392
|
| 132 |
+
},
|
| 133 |
+
{
|
| 134 |
+
"backend": "nemo",
|
| 135 |
+
"success": "5/5",
|
| 136 |
+
"exact_first_word_text": "5/5",
|
| 137 |
+
"first_word_end_mae_s": 0.2210,
|
| 138 |
+
"first_word_end_median_s": 0.0100,
|
| 139 |
+
"first_word_end_max_s": 0.9200,
|
| 140 |
+
"observed_runtime_s": 13.7050
|
| 141 |
+
}
|
| 142 |
+
],
|
| 143 |
+
"full_36_pairwise_material_rows_at_0_250_s": {
|
| 144 |
+
"mfa_vs_whisperx": 7,
|
| 145 |
+
"mfa_vs_nemo": 28,
|
| 146 |
+
"whisperx_vs_nemo": 32
|
| 147 |
+
},
|
| 148 |
+
"canonical_output": {
|
| 149 |
+
"path": "results/final/final-annotated-36.csv",
|
| 150 |
+
"sha256": "37bcb85fc22fb142dae40453b90dd3b427bbfd657861bee7529cd433915535a8"
|
| 151 |
+
}
|
| 152 |
+
},
|
| 153 |
+
"documentation": {
|
| 154 |
+
"readme": "README.md",
|
| 155 |
+
"architecture": "ARCHITECTURE.md",
|
| 156 |
+
"configuration": "docs/CONFIGURATION.md",
|
| 157 |
+
"evaluation": "docs/EVALUATION.md",
|
| 158 |
+
"workflows": "docs/WORKFLOWS.md",
|
| 159 |
+
"testing": "docs/TESTING.md",
|
| 160 |
+
"status": "docs/PROJECT_STATUS.md"
|
| 161 |
+
}
|
| 162 |
+
}
|
|
@@ -1,187 +1,5 @@
|
|
| 1 |
# Architecture
|
| 2 |
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
|
| 6 |
-
|
| 7 |
-
The mandatory design rules are defined in the
|
| 8 |
-
[modular architecture contract](MODULAR_ARCHITECTURE_CONTRACT.md) and the
|
| 9 |
-
[aligner engineering contract](ALIGNER_ENGINEERING_CONTRACT.md).
|
| 10 |
-
|
| 11 |
-
## Layers and Dependency Direction
|
| 12 |
-
|
| 13 |
-
```text
|
| 14 |
-
CLI --------------------> Application --------------------> Core
|
| 15 |
-
| ^
|
| 16 |
-
+----------> Data -------------+
|
| 17 |
-
+----------> Benchmarking -----+
|
| 18 |
-
+----------> Runtime ----------+
|
| 19 |
-
|
| 20 |
-
CLI -- lazy composition --> Concrete Backends
|
| 21 |
-
| |
|
| 22 |
-
+------------> Core
|
| 23 |
-
+------------> Runtime
|
| 24 |
-
```
|
| 25 |
-
|
| 26 |
-
In compact form:
|
| 27 |
-
|
| 28 |
-
```text
|
| 29 |
-
CLI -> Application -> Core
|
| 30 |
-
-> Data
|
| 31 |
-
-> Benchmarking
|
| 32 |
-
-> Runtime
|
| 33 |
-
|
| 34 |
-
Concrete Backends -> Core / Runtime
|
| 35 |
-
```
|
| 36 |
-
|
| 37 |
-
The enforced consequences are:
|
| 38 |
-
|
| 39 |
-
- Core does not import application, data, benchmarking, CLI, or concrete backends.
|
| 40 |
-
- Data and benchmarking do not depend on concrete backends.
|
| 41 |
-
- Application services receive an `AlignmentBackend` protocol instance; they do not
|
| 42 |
-
branch on NeMo, MFA, or WhisperX names.
|
| 43 |
-
- Concrete backend packages do not import one another.
|
| 44 |
-
- The registry stores lazy dotted factories so unavailable optional runtimes do not
|
| 45 |
-
break backend-neutral imports or sibling backends.
|
| 46 |
-
- The CLI delegates annotation and benchmarking to reusable Python services.
|
| 47 |
-
|
| 48 |
-
## Current Package Tree
|
| 49 |
-
|
| 50 |
-
```text
|
| 51 |
-
src/speech_text_data_aligners/
|
| 52 |
-
|-- __init__.py public backend-neutral facade
|
| 53 |
-
|-- application/
|
| 54 |
-
| |-- annotate.py annotation orchestration and publication policy
|
| 55 |
-
| `-- benchmark.py identical-request benchmark orchestration
|
| 56 |
-
|-- backends/
|
| 57 |
-
| |-- registry.py immutable lazy backend composition
|
| 58 |
-
| |-- nemo/
|
| 59 |
-
| | |-- backend.py neutral adapter
|
| 60 |
-
| | |-- config.py typed configuration
|
| 61 |
-
| | |-- manifest.py reference JSONL and output-manifest parsing
|
| 62 |
-
| | |-- runner.py isolated NFA execution
|
| 63 |
-
| | `-- ctm.py word CTM parsing and mapping
|
| 64 |
-
| |-- mfa/
|
| 65 |
-
| | |-- backend.py neutral adapter
|
| 66 |
-
| | |-- config.py typed configuration
|
| 67 |
-
| | |-- corpus.py temporary WAV/LAB corpus
|
| 68 |
-
| | |-- runner.py isolated MFA execution
|
| 69 |
-
| | `-- textgrid.py word-tier parsing and mapping
|
| 70 |
-
| `-- whisperx/
|
| 71 |
-
| |-- backend.py neutral adapter
|
| 72 |
-
| |-- config.py typed configuration
|
| 73 |
-
| |-- segments.py full-WAV authoritative segments
|
| 74 |
-
| |-- provider.py isolated-worker transport
|
| 75 |
-
| |-- worker.py lazy heavyweight native runtime
|
| 76 |
-
| `-- parser.py native word-dictionary parsing and mapping
|
| 77 |
-
|-- benchmarking/
|
| 78 |
-
| |-- metrics.py sparse-gold and pairwise metrics
|
| 79 |
-
| `-- reports.py JSON/CSV/Markdown report rendering
|
| 80 |
-
|-- cli/
|
| 81 |
-
| |-- main.py command dispatch
|
| 82 |
-
| |-- annotate.py annotation argument adapter
|
| 83 |
-
| `-- benchmark.py benchmark argument adapter
|
| 84 |
-
|-- core/
|
| 85 |
-
| |-- models.py immutable neutral values and provenance
|
| 86 |
-
| |-- protocols.py injected backend contracts
|
| 87 |
-
| |-- errors.py project exception taxonomy
|
| 88 |
-
| |-- normalization.py lexical plans and monotonic reconciliation
|
| 89 |
-
| |-- validation.py result and publication-path invariants
|
| 90 |
-
| `-- extraction.py six first/last output fields
|
| 91 |
-
|-- data/
|
| 92 |
-
| |-- audio.py WAV resolution and lightweight probing
|
| 93 |
-
| `-- metadata.py preserving, atomic CSV I/O
|
| 94 |
-
`-- runtime/
|
| 95 |
-
|-- process.py shell-free injected subprocess boundary
|
| 96 |
-
|-- artifacts.py temporary and retained artifact lifecycle
|
| 97 |
-
`-- receipts.py hashes and run provenance
|
| 98 |
-
```
|
| 99 |
-
|
| 100 |
-
Package-marker `__init__.py` files not shown above are deliberately empty. The root
|
| 101 |
-
`__init__.py` is the tested public facade.
|
| 102 |
-
|
| 103 |
-
## Backend Isolation and Native Containment
|
| 104 |
-
|
| 105 |
-
```text
|
| 106 |
-
PARENT PROCESS
|
| 107 |
-
|
| 108 |
-
Python caller or CLI
|
| 109 |
-
|
|
| 110 |
-
v
|
| 111 |
-
backend-neutral application API
|
| 112 |
-
|
|
| 113 |
-
v
|
| 114 |
-
AlignmentBackend protocol
|
| 115 |
-
|
|
| 116 |
-
v
|
| 117 |
-
concrete adapter (NeMo | MFA | WhisperX)
|
| 118 |
-
|
|
| 119 |
-
| injected process/provider boundary
|
| 120 |
-
v
|
| 121 |
-
+------------------------------------------------------+
|
| 122 |
-
| ISOLATED NATIVE RUNTIME |
|
| 123 |
-
| NeMo NFA process | MFA process | WhisperX worker |
|
| 124 |
-
| Heavy optional dependencies and model lifecycle |
|
| 125 |
-
+------------------------------------------------------+
|
| 126 |
-
|
|
| 127 |
-
| native files / native dictionaries
|
| 128 |
-
v
|
| 129 |
-
adapter-local builder, parser, and reconciliation
|
| 130 |
-
|
|
| 131 |
-
| only neutral records cross this point
|
| 132 |
-
v
|
| 133 |
-
AlignmentResult | AlignmentFailure | diagnostics | provenance
|
| 134 |
-
```
|
| 135 |
-
|
| 136 |
-
NeMo manifests, output manifests, and CTM rows remain inside
|
| 137 |
-
`backends.nemo`. MFA corpus labels and TextGrid objects remain inside
|
| 138 |
-
`backends.mfa`. WhisperX segment payloads and native result dictionaries remain
|
| 139 |
-
inside `backends.whisperx`. NeMo CTM, MFA TextGrid, and WhisperX native dictionaries
|
| 140 |
-
never escape their adapters; callers receive only backend-neutral models.
|
| 141 |
-
|
| 142 |
-
The parent process does not import heavyweight NeMo, MFA, WhisperX, Torch, or
|
| 143 |
-
Torchaudio packages. NeMo and MFA run through the shared shell-free process boundary.
|
| 144 |
-
WhisperX runs in a dedicated worker that imports its heavy stack only after invocation.
|
| 145 |
-
|
| 146 |
-
## Transcript-to-Result Data Flow
|
| 147 |
-
|
| 148 |
-
```text
|
| 149 |
-
authoritative transcript
|
| 150 |
-
|
|
| 151 |
-
v
|
| 152 |
-
backend-specific reference representation
|
| 153 |
-
|
|
| 154 |
-
v
|
| 155 |
-
native timed words
|
| 156 |
-
|
|
| 157 |
-
v
|
| 158 |
-
shared monotonic reconciliation
|
| 159 |
-
|
|
| 160 |
-
v
|
| 161 |
-
shared validation
|
| 162 |
-
|
|
| 163 |
-
v
|
| 164 |
-
AlignmentResult
|
| 165 |
-
```
|
| 166 |
-
|
| 167 |
-
The authoritative `en_transcript` remains unchanged throughout. A backend may
|
| 168 |
-
normalize or expand text only through a reversible plan that retains ownership of
|
| 169 |
-
original lexical words. Shared reconciliation requires complete monotonic coverage,
|
| 170 |
-
and shared validation rejects non-finite, negative, reversed, overlapping, incomplete,
|
| 171 |
-
or out-of-audio timestamps. It never clamps, interpolates, or invents a timestamp.
|
| 172 |
-
|
| 173 |
-
## Reuse and Composition
|
| 174 |
-
|
| 175 |
-
The root facade exposes reusable annotation and benchmark services. Applications may
|
| 176 |
-
inject any object satisfying `AlignmentBackend`, including a lightweight test or
|
| 177 |
-
custom backend. The default registry is a convenience for the CLI, not a dependency of
|
| 178 |
-
the application layer.
|
| 179 |
-
|
| 180 |
-
Benchmarking consumes the same neutral results as annotation. Native parsing,
|
| 181 |
-
transcript mapping, CSV I/O, metric calculation, report rendering, and process
|
| 182 |
-
execution each have one owning layer, so adding another adapter does not require
|
| 183 |
-
changes to established backend-neutral behavior.
|
| 184 |
-
|
| 185 |
-
Architecture is enforced by deterministic AST tests and Import Linter contracts. The
|
| 186 |
-
accepted audit and complete dependency evidence are recorded in the
|
| 187 |
-
[Stage 5 final acceptance report](STAGE_5_FINAL_ACCEPTANCE_REPORT.md).
|
|
|
|
| 1 |
# Architecture
|
| 2 |
|
| 3 |
+
The current architecture document is maintained at
|
| 4 |
+
[`../ARCHITECTURE.md`](../ARCHITECTURE.md). This path remains available for older
|
| 5 |
+
links and historical reports.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@@ -0,0 +1,282 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Configuration
|
| 2 |
+
|
| 3 |
+
This page is the authoritative configuration reference for Speech-Text Data Aligners.
|
| 4 |
+
Typed backend dataclasses validate configuration before model or process execution.
|
| 5 |
+
Unknown keys and invalid primitive types are rejected rather than ignored.
|
| 6 |
+
|
| 7 |
+
## Manifest Versus Runtime Configuration
|
| 8 |
+
|
| 9 |
+
[`config.json`](../config.json) is a versioned machine-readable project/model manifest.
|
| 10 |
+
It records package identity, the data contract, registered backends, pinned model
|
| 11 |
+
identity, evaluation scope, and documentation paths. It is validated by
|
| 12 |
+
`tests/quality/test_config_manifest.py`.
|
| 13 |
+
|
| 14 |
+
The manifest is not passed directly to `--backend-config-json`. Runtime configuration
|
| 15 |
+
has these CLI shapes:
|
| 16 |
+
|
| 17 |
+
| Command | `--backend-config-json` shape |
|
| 18 |
+
|---|---|
|
| 19 |
+
| `annotate` | One selected backend's configuration object, for example `{"environment_name":"speech-aligners-mfa"}` |
|
| 20 |
+
| `benchmark` | One outer object keyed by each selected backend, for example `{"mfa":{...},"whisperx":{...}}` |
|
| 21 |
+
| `readiness` | Provider configuration lives under each catalog `backends.<name>.config` value |
|
| 22 |
+
|
| 23 |
+
All path-valued JSON strings are converted to `pathlib.Path`. `null` is accepted only
|
| 24 |
+
for fields documented as optional or nullable.
|
| 25 |
+
|
| 26 |
+
## Environment Isolation
|
| 27 |
+
|
| 28 |
+
| Runtime | Specification | Default selector | Notes |
|
| 29 |
+
|---|---|---|---|
|
| 30 |
+
| Core development | `environments/core-dev.yml` | `speech-aligners-core-dev` | Deterministic tests, lint, type, coverage, and import checks |
|
| 31 |
+
| MFA | `environments/mfa.yml` | `speech-aligners-mfa` | Dedicated CPU Conda runtime |
|
| 32 |
+
| NeMo | `environments/nemo.yml` | No default environment name | Requires explicit pinned source/script and local or registry model |
|
| 33 |
+
| WhisperX | `environments/whisperx.yml` | Current Python by default | Dedicated CPU runtime is recommended |
|
| 34 |
+
|
| 35 |
+
MFA and NeMo execute through an isolated process boundary. WhisperX executes in a
|
| 36 |
+
dedicated worker that imports Torch/Torchaudio/WhisperX only after invocation. Heavy
|
| 37 |
+
backend packages are never imported by core, data, benchmarking, or application layers.
|
| 38 |
+
|
| 39 |
+
## Common Artifact Policy
|
| 40 |
+
|
| 41 |
+
MFA, NeMo, and WhisperX share these artifact semantics:
|
| 42 |
+
|
| 43 |
+
| `artifact_policy` | Behavior |
|
| 44 |
+
|---|---|
|
| 45 |
+
| `never` | Remove adapter workspaces; default |
|
| 46 |
+
| `on_failure` | Retain native workspace only after failure |
|
| 47 |
+
| `always` | Retain native workspace on success or failure |
|
| 48 |
+
|
| 49 |
+
When the policy is not `never`, `retention_root` is required. `run_id` must be one safe
|
| 50 |
+
path component. `temporary_root` is an optional parent for unique workspaces.
|
| 51 |
+
|
| 52 |
+
Retained native artifacts are diagnostic evidence. They do not replace the
|
| 53 |
+
backend-neutral result, and native CTM/TextGrid/WhisperX structures do not cross the
|
| 54 |
+
adapter interface.
|
| 55 |
+
|
| 56 |
+
## MFA Configuration
|
| 57 |
+
|
| 58 |
+
Python type: `speech_text_data_aligners.backends.mfa.config.MfaConfig`.
|
| 59 |
+
|
| 60 |
+
| Field | Type | Default | Meaning |
|
| 61 |
+
|---|---|---|---|
|
| 62 |
+
| `conda_executable` | string | `conda` | Conda command or executable path |
|
| 63 |
+
| `environment_name` | string/null | `speech-aligners-mfa` | Dedicated environment name |
|
| 64 |
+
| `environment_prefix` | path/null | `null` | Dedicated environment prefix |
|
| 65 |
+
| `mfa_executable` | string | `mfa` | Executable inside the selected environment |
|
| 66 |
+
| `model_id` | string | `MontrealCorpusTools/english_mfa` | Hugging Face MFA model repository, without revision |
|
| 67 |
+
| `model_revision` | string | `1009e3c6c689f10875146a5e4e5c2233f8eaf922` | Required 40-character lowercase commit |
|
| 68 |
+
| `model_version` | string | `3.3.0` | Model bundle version recorded in provenance |
|
| 69 |
+
| `dialect` | string | `english_us_mfa` | Dictionary/G2P dialect |
|
| 70 |
+
| `use_g2p` | boolean | `true` | Generate pronunciations for dictionary OOVs |
|
| 71 |
+
| `expected_mfa_version` | string | `3.4.2` | Required MFA distribution |
|
| 72 |
+
| `expected_kaldi_version` | string | `5.5.1172` | Required Kaldi distribution |
|
| 73 |
+
| `expected_kalpy_version` | string | `0.10.5` | Required Kalpy distribution |
|
| 74 |
+
| `expected_openfst_version` | string | `1.8.4` | Required OpenFst distribution |
|
| 75 |
+
| `expected_model_sha256` | string/null | `null` | Optional lowercase model-tree SHA-256 |
|
| 76 |
+
| `num_jobs` | positive integer | `1` | MFA worker process count |
|
| 77 |
+
| `timeout_s` | positive finite number | `3600.0` | Timeout for probe and alignment commands |
|
| 78 |
+
| `artifact_policy` | enum | `never` | Native artifact retention policy |
|
| 79 |
+
| `temporary_root` | path/null | `null` | Parent for temporary workspaces |
|
| 80 |
+
| `retention_root` | path/null | `null` | Required parent when retention is enabled |
|
| 81 |
+
| `run_id` | string | `mfa-run` | Safe retained-workspace component |
|
| 82 |
+
|
| 83 |
+
Exactly one of `environment_name` and `environment_prefix` must be set. When selecting
|
| 84 |
+
a prefix from CLI JSON, use `"environment_name": null` explicitly. The adapter always
|
| 85 |
+
uses long TextGrid output and a commit-qualified model; those safety choices are not
|
| 86 |
+
configurable.
|
| 87 |
+
|
| 88 |
+
Minimal CLI JSON:
|
| 89 |
+
|
| 90 |
+
```json
|
| 91 |
+
{
|
| 92 |
+
"environment_name": "speech-aligners-mfa",
|
| 93 |
+
"num_jobs": 1,
|
| 94 |
+
"artifact_policy": "never"
|
| 95 |
+
}
|
| 96 |
+
```
|
| 97 |
+
|
| 98 |
+
Failure-only artifact example:
|
| 99 |
+
|
| 100 |
+
```json
|
| 101 |
+
{
|
| 102 |
+
"environment_prefix": "/opt/conda/envs/speech-aligners-mfa",
|
| 103 |
+
"environment_name": null,
|
| 104 |
+
"artifact_policy": "on_failure",
|
| 105 |
+
"retention_root": "artifacts/mfa",
|
| 106 |
+
"run_id": "metadata-run-001"
|
| 107 |
+
}
|
| 108 |
+
```
|
| 109 |
+
|
| 110 |
+
## NeMo / NFA Configuration
|
| 111 |
+
|
| 112 |
+
Python type: `speech_text_data_aligners.backends.nemo.config.NemoConfig`.
|
| 113 |
+
|
| 114 |
+
| Field | Type | Default | Meaning |
|
| 115 |
+
|---|---|---|---|
|
| 116 |
+
| `align_script` | path | Required | Exact `tools/nemo_forced_aligner/align.py` |
|
| 117 |
+
| `python_executable` | path | Current interpreter | Python in the isolated NeMo runtime |
|
| 118 |
+
| `pretrained_name` | string/null | `stt_en_fastconformer_hybrid_large_pc` | Registry model name |
|
| 119 |
+
| `model_path` | path/null | `null` | Local `.nemo` checkpoint |
|
| 120 |
+
| `model_id` | string | `stt_en_fastconformer_hybrid_large_pc` | Stable model identity for provenance |
|
| 121 |
+
| `model_revision` | string/null | `null` | Registry/source revision when known |
|
| 122 |
+
| `expected_model_sha256` | string/null | `null` | Required local checkpoint SHA-256 |
|
| 123 |
+
| `expected_nemo_version` | string | `3.0.0` | Required NeMo distribution |
|
| 124 |
+
| `nemo_source_revision` | string | `fd6a877539710e2b98f28c43272ff81312f83417` | Pinned Speech checkout |
|
| 125 |
+
| `expected_align_script_sha256` | string | `1cfaf9526b47037204b7874c4432e4d5d07364a7fca1b840052c370d46ee2f03` | Required entry-point hash |
|
| 126 |
+
| `transcribe_device` | string | `cuda` | Torch device for acoustic inference |
|
| 127 |
+
| `viterbi_device` | string | `cuda` | Torch device for Viterbi alignment |
|
| 128 |
+
| `batch_size` | positive integer | `1` | NFA batch size |
|
| 129 |
+
| `use_local_attention` | boolean | `true` | Request supported local attention |
|
| 130 |
+
| `timeout_s` | positive finite number | `3600.0` | Timeout for probe and alignment |
|
| 131 |
+
| `save_ass` | boolean | `false` | Retain optional ASS output in addition to CTM |
|
| 132 |
+
| `artifact_policy` | enum | `never` | Native artifact retention policy |
|
| 133 |
+
| `temporary_root` | path/null | `null` | Parent for temporary workspaces |
|
| 134 |
+
| `retention_root` | path/null | `null` | Required parent when retention is enabled |
|
| 135 |
+
| `run_id` | string | `nemo-run` | Safe retained-workspace component |
|
| 136 |
+
|
| 137 |
+
Exactly one of `pretrained_name` and `model_path` must be set. When using a local
|
| 138 |
+
checkpoint from CLI JSON, set `"pretrained_name": null` explicitly.
|
| 139 |
+
`expected_model_sha256` is valid only with `model_path`.
|
| 140 |
+
|
| 141 |
+
Local-checkpoint example:
|
| 142 |
+
|
| 143 |
+
```json
|
| 144 |
+
{
|
| 145 |
+
"align_script": "/opt/runtimes/nemo-speech-v3.0.0/tools/nemo_forced_aligner/align.py",
|
| 146 |
+
"python_executable": "/opt/runtimes/nemo-speech-v3.0.0/.venv/bin/python",
|
| 147 |
+
"pretrained_name": null,
|
| 148 |
+
"model_path": "/opt/models/stt_en_fastconformer_hybrid_large_pc.nemo",
|
| 149 |
+
"expected_model_sha256": "<lowercase-64-character-sha256>",
|
| 150 |
+
"transcribe_device": "cuda",
|
| 151 |
+
"viterbi_device": "cuda",
|
| 152 |
+
"artifact_policy": "on_failure",
|
| 153 |
+
"retention_root": "artifacts/nemo",
|
| 154 |
+
"run_id": "metadata-run-001"
|
| 155 |
+
}
|
| 156 |
+
```
|
| 157 |
+
|
| 158 |
+
Reference-text mode, CTM output, and unmodified zero-minimum timestamps are mandatory
|
| 159 |
+
and intentionally not configurable.
|
| 160 |
+
|
| 161 |
+
## WhisperX Configuration
|
| 162 |
+
|
| 163 |
+
Python type: `speech_text_data_aligners.backends.whisperx.config.WhisperXConfig`.
|
| 164 |
+
|
| 165 |
+
| Field | Type | Default | Meaning |
|
| 166 |
+
|---|---|---|---|
|
| 167 |
+
| `python_executable` | path | Current interpreter | Python in the isolated WhisperX runtime |
|
| 168 |
+
| `model_id` | string | `WAV2VEC2_ASR_BASE_960H` | English alignment bundle |
|
| 169 |
+
| `model_revision` | string | `torchaudio-2.8.0` | Model asset revision recorded in provenance |
|
| 170 |
+
| `expected_model_sha256` | string/null | `null` | Optional model-cache aggregate hash |
|
| 171 |
+
| `language` | string | `en` | Alignment language; only `en` is supported |
|
| 172 |
+
| `device` | string | `cpu` | Torch model/alignment device |
|
| 173 |
+
| `model_dir` | path/null | `null` | Persistent model cache directory |
|
| 174 |
+
| `model_cache_only` | boolean | `false` | Disable remote model access where supported |
|
| 175 |
+
| `expected_whisperx_version` | string | `3.8.6` | Required WhisperX distribution |
|
| 176 |
+
| `expected_torch_version` | string | `2.8.0` | Required Torch base version |
|
| 177 |
+
| `expected_torchaudio_version` | string | `2.8.0` | Required Torchaudio base version |
|
| 178 |
+
| `timeout_s` | positive finite number | `3600.0` | Worker-process timeout |
|
| 179 |
+
| `artifact_policy` | enum | `never` | Native artifact retention policy |
|
| 180 |
+
| `temporary_root` | path/null | `null` | Parent for temporary workspaces |
|
| 181 |
+
| `retention_root` | path/null | `null` | Required parent when retention is enabled |
|
| 182 |
+
| `run_id` | string | `whisperx-run` | Safe retained-workspace component |
|
| 183 |
+
|
| 184 |
+
`expected_model_sha256` requires `model_dir`. WhisperX 3.8.6 does not enforce
|
| 185 |
+
`model_cache_only` for Torchaudio bundle downloads, so prepare and hash the cache before
|
| 186 |
+
offline use when remote access must be impossible.
|
| 187 |
+
|
| 188 |
+
CPU example:
|
| 189 |
+
|
| 190 |
+
```json
|
| 191 |
+
{
|
| 192 |
+
"python_executable": "/opt/conda/envs/speech-aligners-whisperx/bin/python",
|
| 193 |
+
"device": "cpu",
|
| 194 |
+
"model_dir": "/opt/models/whisperx-align",
|
| 195 |
+
"model_cache_only": true,
|
| 196 |
+
"expected_model_sha256": "<lowercase-64-character-sha256>"
|
| 197 |
+
}
|
| 198 |
+
```
|
| 199 |
+
|
| 200 |
+
The authoritative transcript, full-duration reference segment, disabled interpolation,
|
| 201 |
+
and no-ASR workflow are mandatory and intentionally not configurable.
|
| 202 |
+
|
| 203 |
+
## Benchmark Composition
|
| 204 |
+
|
| 205 |
+
Pass unique backend names to repeated `--backend` options. The configuration object
|
| 206 |
+
must contain at most those names; entries for unselected names are ignored by the CLI
|
| 207 |
+
factory lookup, while malformed selected values fail construction.
|
| 208 |
+
|
| 209 |
+
```bash
|
| 210 |
+
speech-text-aligners benchmark \
|
| 211 |
+
metadata.csv first-word-gold.csv wavs \
|
| 212 |
+
--backend mfa --backend whisperx \
|
| 213 |
+
--backend-config-json '{"mfa":{"environment_name":"speech-aligners-mfa"},"whisperx":{"device":"cpu"}}' \
|
| 214 |
+
--output-base reports/mfa-whisperx
|
| 215 |
+
```
|
| 216 |
+
|
| 217 |
+
For reproducible heterogeneous runs, prefer constructing typed configs in Python and
|
| 218 |
+
calling `benchmark_backends` with an insertion-ordered mapping.
|
| 219 |
+
|
| 220 |
+
## Readiness Catalog
|
| 221 |
+
|
| 222 |
+
The readiness catalog is a JSON object:
|
| 223 |
+
|
| 224 |
+
```json
|
| 225 |
+
{
|
| 226 |
+
"backends": {
|
| 227 |
+
"mfa": {
|
| 228 |
+
"identity": {
|
| 229 |
+
"backend_name": "mfa",
|
| 230 |
+
"backend_version": "3.4.2",
|
| 231 |
+
"model_id": "MontrealCorpusTools/english_mfa",
|
| 232 |
+
"model_revision": "1009e3c6c689f10875146a5e4e5c2233f8eaf922"
|
| 233 |
+
},
|
| 234 |
+
"config": {
|
| 235 |
+
"environment_name": "speech-aligners-mfa"
|
| 236 |
+
}
|
| 237 |
+
},
|
| 238 |
+
"whisperx": {
|
| 239 |
+
"identity": {
|
| 240 |
+
"backend_name": "whisperx",
|
| 241 |
+
"backend_version": "3.8.6",
|
| 242 |
+
"model_id": "WAV2VEC2_ASR_BASE_960H"
|
| 243 |
+
},
|
| 244 |
+
"config": null,
|
| 245 |
+
"unavailable_reason": "runtime is intentionally not installed"
|
| 246 |
+
}
|
| 247 |
+
},
|
| 248 |
+
"evidence": [
|
| 249 |
+
{
|
| 250 |
+
"audio_id": "example_audio_id",
|
| 251 |
+
"backend": "mfa",
|
| 252 |
+
"candidates": [
|
| 253 |
+
{
|
| 254 |
+
"path": "/absolute/path/to/evidence.json",
|
| 255 |
+
"sha256": "<lowercase-64-character-sha256>"
|
| 256 |
+
}
|
| 257 |
+
]
|
| 258 |
+
}
|
| 259 |
+
]
|
| 260 |
+
}
|
| 261 |
+
```
|
| 262 |
+
|
| 263 |
+
`identity.backend_name` must match its map key and the constructed adapter name.
|
| 264 |
+
`config: null` means the runtime is explicitly unavailable. Evidence candidates are
|
| 265 |
+
reused only after audio, transcript, backend/model provenance, and complete full-word
|
| 266 |
+
timing are verified. See [W1-T Semantic Readiness](W1_T_SEMANTIC_READINESS.md).
|
| 267 |
+
|
| 268 |
+
## Provenance Fields
|
| 269 |
+
|
| 270 |
+
For empirical runs, retain at least:
|
| 271 |
+
|
| 272 |
+
- backend and backend package version;
|
| 273 |
+
- model/checkpoint/dictionary identifiers;
|
| 274 |
+
- model revision/hash when feasible;
|
| 275 |
+
- Python and Torch/CUDA versions where relevant;
|
| 276 |
+
- CPU/GPU device;
|
| 277 |
+
- command/configuration identity;
|
| 278 |
+
- input metadata hash;
|
| 279 |
+
- result and diagnostic artifact paths.
|
| 280 |
+
|
| 281 |
+
The runtime receipt and backend result provenance carry these fields. Preserve receipts
|
| 282 |
+
with benchmark outputs; do not infer missing versions from documentation alone.
|
|
@@ -0,0 +1,87 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Documentation Map
|
| 2 |
+
|
| 3 |
+
This repository separates current operating guidance from immutable acceptance and
|
| 4 |
+
empirical history. Start with the README, then use the focused document that matches
|
| 5 |
+
the task.
|
| 6 |
+
|
| 7 |
+
## Current Guidance
|
| 8 |
+
|
| 9 |
+
| Document | Use it for |
|
| 10 |
+
|---|---|
|
| 11 |
+
| [`README.md`](../README.md) | Hugging Face-style overview, install, quick starts, model matrix, headline metrics |
|
| 12 |
+
| [`config.json`](../config.json) | Machine-readable project, schema, model, and evaluation manifest |
|
| 13 |
+
| [`ARCHITECTURE.md`](../ARCHITECTURE.md) | Current layers, modules, data flow, isolation, and evolution rules |
|
| 14 |
+
| [`CONFIGURATION.md`](CONFIGURATION.md) | Typed fields, defaults, invariants, CLI JSON, readiness catalogs |
|
| 15 |
+
| [`EVALUATION.md`](EVALUATION.md) | Compared configurations, metrics, results, limitations, artifacts |
|
| 16 |
+
| [`WORKFLOWS.md`](WORKFLOWS.md) | End-to-end annotation, benchmark, backend, and readiness flows |
|
| 17 |
+
| [`TESTING.md`](TESTING.md) | Quality commands, markers, coverage, and focused-test map |
|
| 18 |
+
| [`PROJECT_STATUS.md`](PROJECT_STATUS.md) | Accepted operational status and evidence boundary |
|
| 19 |
+
| [`W1_T_SEMANTIC_READINESS.md`](W1_T_SEMANTIC_READINESS.md) | Specialized offline semantic-boundary timing workflow |
|
| 20 |
+
|
| 21 |
+
## Contracts
|
| 22 |
+
|
| 23 |
+
| Document | Authority |
|
| 24 |
+
|---|---|
|
| 25 |
+
| [`ALIGNER_ENGINEERING_CONTRACT.md`](ALIGNER_ENGINEERING_CONTRACT.md) | Data columns, first/last semantics, reference-text and timestamp invariants |
|
| 26 |
+
| [`MODULAR_ARCHITECTURE_CONTRACT.md`](MODULAR_ARCHITECTURE_CONTRACT.md) | Mandatory layering, dependency direction, reuse, isolation, and coverage |
|
| 27 |
+
| [`AGENTS.md`](../AGENTS.md) | Repository execution and acceptance rules for coding agents |
|
| 28 |
+
| [`SKILLS.md`](../SKILLS.md) | Reusable implementation and verification patterns |
|
| 29 |
+
|
| 30 |
+
Contracts change only when the implementation/API changes intentionally. Update the
|
| 31 |
+
contract and its focused tests together.
|
| 32 |
+
|
| 33 |
+
## Decision and Empirical Evidence
|
| 34 |
+
|
| 35 |
+
| Document/artifact | Authority |
|
| 36 |
+
|---|---|
|
| 37 |
+
| [`FINAL_ALIGNER_SELECTION.md`](FINAL_ALIGNER_SELECTION.md) | Current operational selection and limitations |
|
| 38 |
+
| [`POSTFINAL_THREE_ALIGNER_COMPARISON.md`](POSTFINAL_THREE_ALIGNER_COMPARISON.md) | Symmetric three-backend full-36 comparison |
|
| 39 |
+
| `results/final/` | Canonical selection and MFA annotated output |
|
| 40 |
+
| `results/postfinal/` | Full-36 MFA extension and three-backend evidence |
|
| 41 |
+
| `results/w1t/` | Readiness timing, consensus, review, and receipt artifacts |
|
| 42 |
+
|
| 43 |
+
Evidence files are preserved as reviewed. Documentation may summarize and link them;
|
| 44 |
+
it must not rewrite hashes, source cells, timing, or historical conclusions.
|
| 45 |
+
|
| 46 |
+
## Historical Reports
|
| 47 |
+
|
| 48 |
+
| Document | Historical scope |
|
| 49 |
+
|---|---|
|
| 50 |
+
| [`STAGE_0_REPORT.md`](STAGE_0_REPORT.md) | Research, dependency selection, and original architecture freeze |
|
| 51 |
+
| [`STAGE_1_REPORT.md`](STAGE_1_REPORT.md) | Shared framework implementation and closure repairs |
|
| 52 |
+
| [`STAGE_2_REPORT.md`](STAGE_2_REPORT.md) | NeMo adapter implementation and original empirical boundary |
|
| 53 |
+
| [`STAGE_3_REPORT.md`](STAGE_3_REPORT.md) | MFA adapter and original five-row evidence |
|
| 54 |
+
| [`STAGE_4_REPORT.md`](STAGE_4_REPORT.md) | WhisperX adapter and original five-row/full-run evidence |
|
| 55 |
+
| [`STAGE_5_FINAL_ACCEPTANCE_REPORT.md`](STAGE_5_FINAL_ACCEPTANCE_REPORT.md) | Original software/architecture acceptance snapshot |
|
| 56 |
+
| [`POST_STAGE5_DOCS_LICENSE_REPORT.md`](POST_STAGE5_DOCS_LICENSE_REPORT.md) | Documentation, headers, and licensing audit |
|
| 57 |
+
|
| 58 |
+
Stage reports are provenance snapshots. Their counts, paths, and selection statements
|
| 59 |
+
may be superseded by later evidence. Current instructions live in the documents above.
|
| 60 |
+
|
| 61 |
+
## Maintenance Rules
|
| 62 |
+
|
| 63 |
+
When behavior or evidence changes:
|
| 64 |
+
|
| 65 |
+
1. Update the current focused guide first.
|
| 66 |
+
2. Update README/config only for user-facing identity, defaults, models, or headline
|
| 67 |
+
metrics.
|
| 68 |
+
3. Add or update a result receipt/evidence artifact before citing a new empirical
|
| 69 |
+
claim.
|
| 70 |
+
4. Keep historical reports unchanged unless correcting a factual link or marking a
|
| 71 |
+
superseding record.
|
| 72 |
+
5. Validate all local Markdown links and repository paths.
|
| 73 |
+
6. Run every documented syntax/help/quality command before finalizing.
|
| 74 |
+
7. Update `.agents/plans`, `.agents/work_data`, and `.agents/logs` with decisions and
|
| 75 |
+
observed results.
|
| 76 |
+
8. Review `AGENTS.md` and `SKILLS.md` at the end for durable process gaps.
|
| 77 |
+
|
| 78 |
+
## Terminology
|
| 79 |
+
|
| 80 |
+
- **Authoritative transcript**: the original `en_transcript` cell.
|
| 81 |
+
- **Forced alignment**: locating supplied text in audio; not ASR transcription.
|
| 82 |
+
- **Manual gold**: human reference timing/text fields.
|
| 83 |
+
- **Pairwise disagreement**: difference between two backend outputs; not accuracy.
|
| 84 |
+
- **Structural success**: valid complete word/timing output; not empirical accuracy.
|
| 85 |
+
- **Observed runtime**: one run's measured time under its stated hardware/cache state.
|
| 86 |
+
- **Current guidance**: maintained operating documentation.
|
| 87 |
+
- **Historical report**: preserved stage/acceptance snapshot.
|
|
@@ -0,0 +1,195 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Evaluation
|
| 2 |
+
|
| 3 |
+
This page is the canonical comparison of the alignment models and configurations in
|
| 4 |
+
this repository. It distinguishes manual-gold accuracy, structural success, pairwise
|
| 5 |
+
disagreement, diagnostic behavior, runtime observations, and offline readiness
|
| 6 |
+
agreement. Those evidence types are not interchangeable.
|
| 7 |
+
|
| 8 |
+
## Evaluation Scope
|
| 9 |
+
|
| 10 |
+
| Evidence set | Rows | Available reference | Purpose |
|
| 11 |
+
|---|---:|---|---|
|
| 12 |
+
| Five-row manual gold | 5 | `first_word_text`, `first_word_end_s` | Limited timing accuracy |
|
| 13 |
+
| Full metadata set | 36 | Authoritative transcripts and WAV bounds | Structural success and cross-backend disagreement |
|
| 14 |
+
| W1-T semantic-readiness timing | 36 | Frozen semantic boundary tokens | Offline automatic END estimates and agreement |
|
| 15 |
+
|
| 16 |
+
The metadata evaluation aligns the supplied `en_transcript`; it does not score ASR
|
| 17 |
+
transcription. The five-row file contains no gold for `first_word_start_s`,
|
| 18 |
+
`last_word_start_s`, or `last_word_end_s`.
|
| 19 |
+
|
| 20 |
+
## Compared Models and Configurations
|
| 21 |
+
|
| 22 |
+
| Backend | Runtime | Model identity | Dictionary/model revision | Device and cache |
|
| 23 |
+
|---|---|---|---|---|
|
| 24 |
+
| MFA | MFA 3.4.2; Kaldi 5.5.1172; Kalpy 0.10.5; OpenFst 1.8.4 | `MontrealCorpusTools/english_mfa` model 3.3.0 | Revision `1009e3c6c689f10875146a5e4e5c2233f8eaf922`; `english_us_mfa`; G2P enabled | CPU; cold/model-cache conditions vary by run |
|
| 25 |
+
| WhisperX | WhisperX 3.8.6; Torch/Torchaudio 2.8.0 | `WAV2VEC2_ASR_BASE_960H` | Torchaudio 2.8.0 bundle; observed asset SHA-256 `488fd4f...c945d` | CPU; prepopulated hash-verified cache in accepted runs |
|
| 26 |
+
| NeMo / NFA | NeMo 3.0.0 source `fd6a877539710e2b98f28c43272ff81312f83417` | `stt_en_fastconformer_hybrid_large_pc` | Local checkpoint hash `37afe662...2c8b61`; no registry revision recorded | NVIDIA A10G in accepted empirical runs |
|
| 27 |
+
|
| 28 |
+
The full unabbreviated identities and hashes are in [`config.json`](../config.json),
|
| 29 |
+
the run receipts, and [Configuration](CONFIGURATION.md).
|
| 30 |
+
|
| 31 |
+
## Metric Definitions
|
| 32 |
+
|
| 33 |
+
| Metric | Definition | Missing/failure behavior |
|
| 34 |
+
|---|---|---|
|
| 35 |
+
| Success | A backend returned a complete result that passed lexical/timestamp validation | Failure is counted and never receives invented timing |
|
| 36 |
+
| Exact first-word text | Output `first_word_text` exactly equals the reference surface | Failed rows cannot match |
|
| 37 |
+
| First-word-end absolute error | `abs(predicted_first_word_end_s - reference_first_word_end_s)` | Empty reference is counted as missing and excluded |
|
| 38 |
+
| MAE | Mean of available first-word-end absolute errors | No scorable reference yields null |
|
| 39 |
+
| Median / maximum | Median and maximum available first-word-end absolute error | Same exclusion as MAE |
|
| 40 |
+
| Pairwise boundary difference | Absolute difference between two successful backend boundaries | Only rows successful in both backends |
|
| 41 |
+
| Material disagreement | Any first/last boundary differs by at least the declared threshold | Diagnostic review queue, not an error rate |
|
| 42 |
+
| Runtime | Measured service/batch execution time | Observed only; not normalized across hardware/cache |
|
| 43 |
+
|
| 44 |
+
Benchmark reports expose success/failure counts, missing-reference counts, exact-text
|
| 45 |
+
rate, first-end MAE/median/maximum, runtime, and per-boundary pairwise MAE. Historical
|
| 46 |
+
comparison artifacts additionally calculate pairwise median, p95, maximum, and
|
| 47 |
+
threshold counts.
|
| 48 |
+
|
| 49 |
+
## Five-Row Manual Gold
|
| 50 |
+
|
| 51 |
+
Evidence date: 2026-09-06 accepted comparison. Sample size: 5.
|
| 52 |
+
|
| 53 |
+
| Backend | Configuration | Success | Exact first text | First-end MAE | Median | Maximum |
|
| 54 |
+
|---|---|---:|---:|---:|---:|---:|
|
| 55 |
+
| MFA | `english_mfa@1009e3c...`, `english_us_mfa` | 5/5 | 5/5 | **0.0162 s** | **0.0110 s** | **0.0300 s** |
|
| 56 |
+
| WhisperX | `WAV2VEC2_ASR_BASE_960H`, CPU | 5/5 | 5/5 | **0.0154 s** | **0.0140 s** | **0.0310 s** |
|
| 57 |
+
| NeMo | `stt_en_fastconformer_hybrid_large_pc`, A10G | 5/5 | 5/5 | **0.2210 s** | **0.0100 s** | **0.9200 s** |
|
| 58 |
+
|
| 59 |
+
Interpretation:
|
| 60 |
+
|
| 61 |
+
- MFA and WhisperX are effectively tied on the only timing gold.
|
| 62 |
+
- WhisperX MAE is lower by 0.0008 s; the sample is too small to rank by this alone.
|
| 63 |
+
- MFA has slightly lower median and maximum error.
|
| 64 |
+
- NeMo has low median error but a 0.920 s maximum on `EN0949`, producing its larger MAE.
|
| 65 |
+
- These results do not establish accuracy for first-word start or either last boundary.
|
| 66 |
+
|
| 67 |
+
## Full-36 Structural Comparison
|
| 68 |
+
|
| 69 |
+
All three backends produced 36/36 candidates. Every accepted candidate preserved
|
| 70 |
+
source cells/order, the exact six-column extension, authoritative first/last lexical
|
| 71 |
+
surfaces, and finite WAV-bounded monotonic timestamps.
|
| 72 |
+
|
| 73 |
+
### Pairwise Absolute Differences
|
| 74 |
+
|
| 75 |
+
| Pair | Boundary | Mean | Median | p95 | Maximum |
|
| 76 |
+
|---|---|---:|---:|---:|---:|
|
| 77 |
+
| MFA / WhisperX | first start | 0.059306 s | 0.051000 s | 0.116750 s | 0.132000 s |
|
| 78 |
+
| MFA / WhisperX | first end | 0.018694 s | 0.015000 s | 0.039000 s | 0.078000 s |
|
| 79 |
+
| MFA / WhisperX | last start | 0.211611 s | 0.064000 s | 1.062500 s | 1.545000 s |
|
| 80 |
+
| MFA / WhisperX | last end | 0.325083 s | 0.128000 s | 1.449750 s | 1.835000 s |
|
| 81 |
+
| MFA / NeMo | first start | 0.665556 s | 0.325000 s | 1.430000 s | 2.120000 s |
|
| 82 |
+
| MFA / NeMo | first end | 0.292500 s | 0.170000 s | 1.250000 s | 1.510000 s |
|
| 83 |
+
| MFA / NeMo | last start | 0.137778 s | 0.160000 s | 0.225000 s | 0.250000 s |
|
| 84 |
+
| MFA / NeMo | last end | 0.104444 s | 0.100000 s | 0.222500 s | 0.260000 s |
|
| 85 |
+
| WhisperX / NeMo | first start | 0.724861 s | 0.371500 s | 1.453000 s | 2.122000 s |
|
| 86 |
+
| WhisperX / NeMo | first end | 0.292639 s | 0.151000 s | 1.252250 s | 1.503000 s |
|
| 87 |
+
| WhisperX / NeMo | last start | 0.259500 s | 0.106000 s | 1.247500 s | 1.795000 s |
|
| 88 |
+
| WhisperX / NeMo | last end | 0.398861 s | 0.202000 s | 1.534750 s | 1.955000 s |
|
| 89 |
+
|
| 90 |
+
p95 uses linear interpolation at rank `(n - 1) * 0.95`.
|
| 91 |
+
|
| 92 |
+
### Material-Disagreement Counts
|
| 93 |
+
|
| 94 |
+
Counts are rows where at least one of the four boundaries reaches the threshold:
|
| 95 |
+
|
| 96 |
+
| Pair | >=0.250 s | >=0.500 s | >=1.000 s |
|
| 97 |
+
|---|---:|---:|---:|
|
| 98 |
+
| MFA / WhisperX | 7 | 6 | 5 |
|
| 99 |
+
| MFA / NeMo | 28 | 17 | 12 |
|
| 100 |
+
| WhisperX / NeMo | 32 | 21 | 16 |
|
| 101 |
+
|
| 102 |
+
MFA and WhisperX are close at first-word boundaries. All seven of their material rows
|
| 103 |
+
are driven by last-word boundaries. These differences identify review cases; they do
|
| 104 |
+
not show which backend is acoustically correct.
|
| 105 |
+
|
| 106 |
+
## Diagnostic Boundary Signals
|
| 107 |
+
|
| 108 |
+
| Signal | MFA | WhisperX | NeMo |
|
| 109 |
+
|---|---:|---:|---:|
|
| 110 |
+
| Exact zero first-word starts | 0/36 | 0/36 | 17/36 |
|
| 111 |
+
| First-word duration p95 | 0.457 s | 0.400 s | 1.560 s |
|
| 112 |
+
| First-word duration maximum | 0.720 s | 0.621 s | 2.160 s |
|
| 113 |
+
| Final word within 50 ms of WAV end | 0/36 | 0/36 | 0/36 |
|
| 114 |
+
|
| 115 |
+
NeMo's repeated zero starts and the `EN0949` first-end outlier are material operational
|
| 116 |
+
warnings. MFA and NeMo have smaller last-boundary disagreement from each other than
|
| 117 |
+
either has with WhisperX on the largest cases. This is still concordance evidence, not
|
| 118 |
+
manual ground truth.
|
| 119 |
+
|
| 120 |
+
## Runtime Observations
|
| 121 |
+
|
| 122 |
+
| Backend | Observed run | Runtime | Conditions |
|
| 123 |
+
|---|---|---:|---|
|
| 124 |
+
| MFA | Accepted original five-row cold run | 274.9569 s | Cold run; later run observed 79.5288 s under different cache/host conditions |
|
| 125 |
+
| WhisperX | Accepted five-row run | 28.5392 s | CPU, prepopulated hash-verified model cache |
|
| 126 |
+
| NeMo | Accepted five-row run | 13.7050 s | NVIDIA A10G after smoke; caches may have been warm |
|
| 127 |
+
|
| 128 |
+
Runtime comparisons are secondary. Host, device, process overhead, model download, and
|
| 129 |
+
cache state were not controlled. Do not convert these observations into a general
|
| 130 |
+
speed ranking.
|
| 131 |
+
|
| 132 |
+
## W1-T Semantic-Readiness Evidence
|
| 133 |
+
|
| 134 |
+
W1-T maps frozen English semantic-boundary tokens to automatic acoustic END estimates.
|
| 135 |
+
It does not measure human Russian onset and does not adjudicate semantic boundaries.
|
| 136 |
+
|
| 137 |
+
| Metric | Observed result |
|
| 138 |
+
|---|---|
|
| 139 |
+
| Cases | 36 |
|
| 140 |
+
| Mapping | 36 EXACT |
|
| 141 |
+
| Confidence | 1 HIGH, 35 MEDIUM |
|
| 142 |
+
| LOW / SINGLE_BACKEND / UNRESOLVED | 0 / 0 / 0 |
|
| 143 |
+
| Cross-backend range p50 | 22.5 ms |
|
| 144 |
+
| Cross-backend range p95 | 86.5 ms |
|
| 145 |
+
| Cross-backend maximum range | 127 ms |
|
| 146 |
+
| Semantic-ready END p50 | 2.1765 s |
|
| 147 |
+
| Semantic-ready END p95 | 3.316625 s |
|
| 148 |
+
|
| 149 |
+
Consensus is the median of valid independent backend END estimates. HIGH requires
|
| 150 |
+
exactly three valid backends and range <=100 ms; MEDIUM requires at least two and range
|
| 151 |
+
<=250 ms; LOW requires at least two and range >250 ms. Agreement is not timing
|
| 152 |
+
accuracy. See [W1-T Semantic Readiness](W1_T_SEMANTIC_READINESS.md) for reuse,
|
| 153 |
+
provenance, mapping, and materialization rules.
|
| 154 |
+
|
| 155 |
+
## Operational Selection
|
| 156 |
+
|
| 157 |
+
MFA is the selected operational backend as of 2026-09-06. The decision combines:
|
| 158 |
+
|
| 159 |
+
1. strict 36/36 eligibility;
|
| 160 |
+
2. a first-word-end MAE effectively tied with WhisperX;
|
| 161 |
+
3. no repeated zero first-word starts;
|
| 162 |
+
4. diagnostic last-boundary support on the largest disputed cases;
|
| 163 |
+
5. reproducible pinned runtime/model identity.
|
| 164 |
+
|
| 165 |
+
This is an operational choice from limited evidence, not a claim of universal timing
|
| 166 |
+
superiority. WhisperX remains the accepted alternative. NeMo remains usable behind the
|
| 167 |
+
same protocol where its pinned GPU runtime/model are available.
|
| 168 |
+
|
| 169 |
+
## Canonical Artifacts
|
| 170 |
+
|
| 171 |
+
| Artifact | Purpose |
|
| 172 |
+
|---|---|
|
| 173 |
+
| `results/final/final-annotated-36.csv` | Canonical MFA annotation, 36/36 |
|
| 174 |
+
| `results/final/final-annotated-36.receipt.json` | Import/final hashes and evidence links |
|
| 175 |
+
| `results/final/final-aligner-selection.{json,md}` | Canonical selection record |
|
| 176 |
+
| `results/postfinal/three-backend-full-36/` | Full comparison, pairwise tables, runtime and systematic analysis |
|
| 177 |
+
| `results/postfinal/mfa-full-36/` | MFA full-run validation and diagnostics |
|
| 178 |
+
| `results/stage5/` | Historical WhisperX full-run and original acceptance evidence |
|
| 179 |
+
| `results/w1t/` | Offline semantic-readiness sidecars and receipts |
|
| 180 |
+
|
| 181 |
+
The canonical CSV SHA-256 is
|
| 182 |
+
`37bcb85fc22fb142dae40453b90dd3b427bbfd657861bee7529cd433915535a8`.
|
| 183 |
+
Repository CSV copies use LF line endings; import receipts preserve the reviewed
|
| 184 |
+
external artifact hash where line endings differed.
|
| 185 |
+
|
| 186 |
+
## Limitations
|
| 187 |
+
|
| 188 |
+
- Manual gold covers only first-word text/end on five rows.
|
| 189 |
+
- First-word start and last-word boundaries have no manual ground truth.
|
| 190 |
+
- Pairwise agreement cannot establish acoustic accuracy.
|
| 191 |
+
- The 0.0008 s MFA/WhisperX MAE difference is not decision-grade.
|
| 192 |
+
- Full-corpus pairwise analysis is diagnostic and threshold-dependent.
|
| 193 |
+
- Runtime observations are not hardware/cache controlled.
|
| 194 |
+
- Accents, broader acoustic conditions, repeated runs, and larger human-annotated sets
|
| 195 |
+
remain outside the current evidence boundary.
|
|
@@ -32,39 +32,50 @@ src/speech_text_data_aligners/
|
|
| 32 |
│ ├── errors.py # project exception taxonomy
|
| 33 |
│ ├── normalization.py # lexical extraction + reversible mappings
|
| 34 |
│ ├── validation.py # alignment/timestamp invariants
|
| 35 |
-
│
|
|
|
|
| 36 |
├── data/
|
| 37 |
│ ├── metadata.py # CSV read/write/preservation contract
|
| 38 |
-
│
|
|
|
|
| 39 |
├── application/
|
| 40 |
│ ├── annotate.py # reusable annotation use case/service
|
| 41 |
-
│
|
|
|
|
|
|
|
| 42 |
├── benchmarking/
|
| 43 |
│ ├── metrics.py # deterministic gold/pairwise metrics
|
| 44 |
-
│
|
|
|
|
| 45 |
├── runtime/
|
| 46 |
│ ├── process.py # injectable subprocess abstraction
|
| 47 |
-
│
|
|
|
|
| 48 |
├── backends/
|
| 49 |
│ ├── nemo/
|
| 50 |
│ │ ├── backend.py # AlignmentBackend adapter/orchestration
|
| 51 |
│ │ ├── config.py # typed NFA config
|
| 52 |
│ │ ├── manifest.py # native input artifact construction
|
|
|
|
| 53 |
│ │ └── ctm.py # native CTM parser/mapping
|
| 54 |
│ ├── mfa/
|
| 55 |
│ │ ├── backend.py
|
| 56 |
│ │ ├── config.py
|
| 57 |
│ │ ├── corpus.py # temporary corpus/native inputs
|
|
|
|
| 58 |
│ │ └── textgrid.py # word-tier parser/mapping
|
| 59 |
│ └── whisperx/
|
| 60 |
│ ├── backend.py
|
| 61 |
│ ├── config.py
|
| 62 |
│ ├── provider.py # injectable model/API call boundary
|
|
|
|
|
|
|
| 63 |
│ └── parser.py # native result conversion/mapping
|
| 64 |
└── cli/
|
| 65 |
├── main.py # composition root only
|
| 66 |
├── annotate.py # argument/presentation adapter
|
| 67 |
-
|
|
|
|
| 68 |
```
|
| 69 |
|
| 70 |
Do not create files solely to imitate this tree. A smaller module set is acceptable when a file would otherwise be trivial, but every retained module must have one clear responsibility and the layer boundaries below remain mandatory. Conversely, do not collapse unrelated concerns into monolithic backend or CLI modules merely to reduce file count.
|
|
@@ -218,3 +229,6 @@ The final report must contain:
|
|
| 218 |
6. confirmation that importing the backend-neutral library succeeds without heavyweight optional backends installed;
|
| 219 |
7. confirmation that each backend can be disabled/absent without breaking the others;
|
| 220 |
8. any deviations from this contract with explicit rationale.
|
|
|
|
|
|
|
|
|
|
|
|
| 32 |
│ ├── errors.py # project exception taxonomy
|
| 33 |
│ ├── normalization.py # lexical extraction + reversible mappings
|
| 34 |
│ ├── validation.py # alignment/timestamp invariants
|
| 35 |
+
│ ├── extraction.py # first/last lexical boundary selection
|
| 36 |
+
│ └── readiness.py # semantic-boundary mapping and consensus
|
| 37 |
├── data/
|
| 38 |
│ ├── metadata.py # CSV read/write/preservation contract
|
| 39 |
+
│ ├── audio.py # audio_id/path resolution + lightweight metadata
|
| 40 |
+
│ └── readiness.py # frozen readiness cohort/hash verification
|
| 41 |
├── application/
|
| 42 |
│ ├── annotate.py # reusable annotation use case/service
|
| 43 |
+
│ ├── benchmark.py # reusable backend evaluation/comparison use case
|
| 44 |
+
│ ├── readiness.py # reuse-first offline readiness orchestration
|
| 45 |
+
│ └── materialize.py # accepted-evidence full-word timing materialization
|
| 46 |
├── benchmarking/
|
| 47 |
│ ├── metrics.py # deterministic gold/pairwise metrics
|
| 48 |
+
│ ├── reports.py # machine/human-readable reporting
|
| 49 |
+
│ └── readiness.py # readiness distributions/sensitivity reporting
|
| 50 |
├── runtime/
|
| 51 |
│ ├── process.py # injectable subprocess abstraction
|
| 52 |
+
│ ├── receipts.py # environment/model/run provenance capture
|
| 53 |
+
│ └── alignment_evidence.py # full-word evidence integrity/reuse verification
|
| 54 |
├── backends/
|
| 55 |
│ ├── nemo/
|
| 56 |
│ │ ├── backend.py # AlignmentBackend adapter/orchestration
|
| 57 |
│ │ ├── config.py # typed NFA config
|
| 58 |
│ │ ├── manifest.py # native input artifact construction
|
| 59 |
+
│ │ ├── runner.py # isolated NFA process execution
|
| 60 |
│ │ └── ctm.py # native CTM parser/mapping
|
| 61 |
│ ├── mfa/
|
| 62 |
│ │ ├── backend.py
|
| 63 |
│ │ ├── config.py
|
| 64 |
│ │ ├── corpus.py # temporary corpus/native inputs
|
| 65 |
+
│ │ ├── runner.py # isolated MFA process execution
|
| 66 |
│ │ └── textgrid.py # word-tier parser/mapping
|
| 67 |
│ └── whisperx/
|
| 68 |
│ ├── backend.py
|
| 69 |
│ ├── config.py
|
| 70 |
│ ├── provider.py # injectable model/API call boundary
|
| 71 |
+
│ ├── worker.py # lazy heavyweight native runtime
|
| 72 |
+
│ ├── segments.py # full-duration authoritative segments
|
| 73 |
│ └── parser.py # native result conversion/mapping
|
| 74 |
└── cli/
|
| 75 |
├── main.py # composition root only
|
| 76 |
├── annotate.py # argument/presentation adapter
|
| 77 |
+
├── benchmark.py # argument/presentation adapter
|
| 78 |
+
└── readiness.py # readiness catalog/composition adapter
|
| 79 |
```
|
| 80 |
|
| 81 |
Do not create files solely to imitate this tree. A smaller module set is acceptable when a file would otherwise be trivial, but every retained module must have one clear responsibility and the layer boundaries below remain mandatory. Conversely, do not collapse unrelated concerns into monolithic backend or CLI modules merely to reduce file count.
|
|
|
|
| 229 |
6. confirmation that importing the backend-neutral library succeeds without heavyweight optional backends installed;
|
| 230 |
7. confirmation that each backend can be disabled/absent without breaking the others;
|
| 231 |
8. any deviations from this contract with explicit rationale.
|
| 232 |
+
|
| 233 |
+
The current maintained evidence is summarized in [`ARCHITECTURE.md`](../ARCHITECTURE.md)
|
| 234 |
+
and the implementation-to-test/coverage map in [`TESTING.md`](TESTING.md).
|
|
@@ -4,6 +4,11 @@ This page summarizes the accepted Stage 0–5 baseline, NVIDIA-host NeMo closure
|
|
| 4 |
post-final three-backend full-36 comparison completed on 2026-09-06. Software acceptance,
|
| 5 |
empirical validation, and operational selection remain distinct.
|
| 6 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 7 |
## Final Completion
|
| 8 |
|
| 9 |
MFA is the selected operational backend. The canonical reviewed 36-row annotation is
|
|
@@ -58,6 +63,21 @@ Manual gold covers only `first_word_text` and `first_word_end_s` on five rows.
|
|
| 58 |
`first_word_start_s`, `last_word_start_s`, and `last_word_end_s` do not have manual ground
|
| 59 |
truth. Pairwise agreement/disagreement is diagnostic evidence, not accuracy.
|
| 60 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 61 |
## Detailed Evidence
|
| 62 |
|
| 63 |
- [Stage 0 research and architecture](STAGE_0_REPORT.md)
|
|
@@ -68,3 +88,5 @@ truth. Pairwise agreement/disagreement is diagnostic evidence, not accuracy.
|
|
| 68 |
- [Stage 5 final acceptance](STAGE_5_FINAL_ACCEPTANCE_REPORT.md)
|
| 69 |
- [Post-final three-aligner comparison](POSTFINAL_THREE_ALIGNER_COMPARISON.md)
|
| 70 |
- [Final aligner selection](FINAL_ALIGNER_SELECTION.md)
|
|
|
|
|
|
|
|
|
| 4 |
post-final three-backend full-36 comparison completed on 2026-09-06. Software acceptance,
|
| 5 |
empirical validation, and operational selection remain distinct.
|
| 6 |
|
| 7 |
+
Current operating guidance is in the [README](../README.md),
|
| 8 |
+
[Configuration](CONFIGURATION.md), [Evaluation](EVALUATION.md), [Workflows](WORKFLOWS.md),
|
| 9 |
+
and [Testing and Quality](TESTING.md). Stage reports below are preserved acceptance
|
| 10 |
+
history rather than the primary user guide.
|
| 11 |
+
|
| 12 |
## Final Completion
|
| 13 |
|
| 14 |
MFA is the selected operational backend. The canonical reviewed 36-row annotation is
|
|
|
|
| 63 |
`first_word_start_s`, `last_word_start_s`, and `last_word_end_s` do not have manual ground
|
| 64 |
truth. Pairwise agreement/disagreement is diagnostic evidence, not accuracy.
|
| 65 |
|
| 66 |
+
## Current Engineering Gate
|
| 67 |
+
|
| 68 |
+
The documentation-review checkout on 2026-10-30 passes the deterministic suite and
|
| 69 |
+
quality tooling. This is a software-quality result, not additional acoustic evidence.
|
| 70 |
+
|
| 71 |
+
| Gate | Current result |
|
| 72 |
+
|---|---|
|
| 73 |
+
| Deterministic tests | 635 passed, 3 heavyweight integration tests skipped |
|
| 74 |
+
| Aggregate line coverage | 98.74% |
|
| 75 |
+
| Individual implementation coverage | All pass; minimum 93.33% |
|
| 76 |
+
| Ruff / strict mypy | Pass |
|
| 77 |
+
| Import Linter | 7 contracts kept |
|
| 78 |
+
|
| 79 |
+
See [Testing and Quality](TESTING.md) for commands and the complete file/test map.
|
| 80 |
+
|
| 81 |
## Detailed Evidence
|
| 82 |
|
| 83 |
- [Stage 0 research and architecture](STAGE_0_REPORT.md)
|
|
|
|
| 88 |
- [Stage 5 final acceptance](STAGE_5_FINAL_ACCEPTANCE_REPORT.md)
|
| 89 |
- [Post-final three-aligner comparison](POSTFINAL_THREE_ALIGNER_COMPARISON.md)
|
| 90 |
- [Final aligner selection](FINAL_ALIGNER_SELECTION.md)
|
| 91 |
+
- [Canonical evaluation summary](EVALUATION.md)
|
| 92 |
+
- [Configuration reference](CONFIGURATION.md)
|
|
@@ -0,0 +1,177 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Testing and Quality
|
| 2 |
+
|
| 3 |
+
This page defines the deterministic release gate and records the current per-file
|
| 4 |
+
coverage map. Ordinary tests do not download models, call remote services, require a
|
| 5 |
+
GPU, or invoke heavyweight aligners.
|
| 6 |
+
|
| 7 |
+
## Environment
|
| 8 |
+
|
| 9 |
+
Use the core development environment or the local equivalent:
|
| 10 |
+
|
| 11 |
+
```bash
|
| 12 |
+
conda activate speech-text-data-aligners
|
| 13 |
+
python -m pip install -e . --no-deps
|
| 14 |
+
```
|
| 15 |
+
|
| 16 |
+
Pinned development tools are in [`requirements.txt`](../requirements.txt) and
|
| 17 |
+
[`environments/core-dev.yml`](../environments/core-dev.yml).
|
| 18 |
+
|
| 19 |
+
## Deterministic Gate
|
| 20 |
+
|
| 21 |
+
Run from the repository root:
|
| 22 |
+
|
| 23 |
+
```bash
|
| 24 |
+
python -m pytest
|
| 25 |
+
ruff check .
|
| 26 |
+
ruff format --check .
|
| 27 |
+
python -m mypy src
|
| 28 |
+
lint-imports
|
| 29 |
+
coverage erase
|
| 30 |
+
coverage run -m pytest
|
| 31 |
+
coverage report --show-missing
|
| 32 |
+
coverage json -o coverage.json
|
| 33 |
+
python -m pytest tests/quality/test_per_file_coverage.py -q
|
| 34 |
+
```
|
| 35 |
+
|
| 36 |
+
`coverage.json` is generated evidence and is ignored by Git. Regenerate it before
|
| 37 |
+
running the per-file gate; a stale file is not accepted as current proof.
|
| 38 |
+
|
| 39 |
+
## Current Observed Gate
|
| 40 |
+
|
| 41 |
+
Captured in this checkout on 2026-10-30 before documentation-only changes to source:
|
| 42 |
+
|
| 43 |
+
| Gate | Result |
|
| 44 |
+
|---|---|
|
| 45 |
+
| Pytest | 635 passed, 3 skipped |
|
| 46 |
+
| Aggregate implementation line coverage | 98.74% |
|
| 47 |
+
| Lowest implementation file | 93.33% |
|
| 48 |
+
| Lowest shared-layer file | 95.65% |
|
| 49 |
+
| Ruff lint/format | Pass |
|
| 50 |
+
| Strict mypy | Pass in 53 source files |
|
| 51 |
+
| Import Linter | 7 kept, 0 broken |
|
| 52 |
+
| Architecture tests | 10 passed |
|
| 53 |
+
| Header/manifest/per-file tests | Pass |
|
| 54 |
+
|
| 55 |
+
The repository gate is at least 90% for every non-empty implementation file and at
|
| 56 |
+
least 90% aggregate. This repository additionally enforces at least 95% for shared
|
| 57 |
+
`core`, `data`, `application`, `benchmarking`, and `runtime` files.
|
| 58 |
+
|
| 59 |
+
## Test Boundaries
|
| 60 |
+
|
| 61 |
+
| Test area | Purpose |
|
| 62 |
+
|---|---|
|
| 63 |
+
| `tests/unit/core` | Normalization, mapping, models, validation, errors, extraction, readiness |
|
| 64 |
+
| `tests/unit/data` | CSV preservation, WAV resolution/probing, readiness cohort verification |
|
| 65 |
+
| `tests/unit/application` | Public use cases, publication, benchmarking, readiness, materialization |
|
| 66 |
+
| `tests/unit/benchmarking` | Metrics, reports, readiness distributions |
|
| 67 |
+
| `tests/unit/runtime` | Process, artifacts, receipts, evidence integrity |
|
| 68 |
+
| `tests/unit/backends` | Configs, native builders/parsers, adapters, registry, isolated providers |
|
| 69 |
+
| `tests/unit/cli` | Argument validation and delegation to application services |
|
| 70 |
+
| `tests/architecture` | Dependency direction and readiness isolation |
|
| 71 |
+
| `tests/quality` | Headers, manifest consistency, per-file coverage |
|
| 72 |
+
| `tests/integration/*` | Opt-in real subprocess/model smoke tests |
|
| 73 |
+
|
| 74 |
+
Integration tests are marked `integration` plus `mfa`, `nemo`, or `whisperx`. They are
|
| 75 |
+
excluded from ordinary confidence unless their runtime and explicit inputs are
|
| 76 |
+
available.
|
| 77 |
+
|
| 78 |
+
## Opt-In Integration
|
| 79 |
+
|
| 80 |
+
MFA:
|
| 81 |
+
|
| 82 |
+
```bash
|
| 83 |
+
RUN_MFA_INTEGRATION=1 \
|
| 84 |
+
MFA_CONDA_PREFIX=/opt/conda/envs/speech-aligners-mfa \
|
| 85 |
+
MFA_SMOKE_WAV=/absolute/path/to/file.wav \
|
| 86 |
+
MFA_SMOKE_TRANSCRIPT='Exact authoritative transcript.' \
|
| 87 |
+
python -m pytest tests/integration/mfa/test_mfa_real.py -m 'integration and mfa' -v
|
| 88 |
+
```
|
| 89 |
+
|
| 90 |
+
WhisperX:
|
| 91 |
+
|
| 92 |
+
```bash
|
| 93 |
+
RUN_WHISPERX_INTEGRATION=1 \
|
| 94 |
+
WHISPERX_PYTHON=/opt/conda/envs/speech-aligners-whisperx/bin/python \
|
| 95 |
+
WHISPERX_SMOKE_WAV=/absolute/path/to/file.wav \
|
| 96 |
+
WHISPERX_SMOKE_TRANSCRIPT='Exact authoritative transcript.' \
|
| 97 |
+
WHISPERX_MODEL_DIR=/opt/models/whisperx-align \
|
| 98 |
+
python -m pytest tests/integration/whisperx/test_whisperx_real.py -m 'integration and whisperx' -v
|
| 99 |
+
```
|
| 100 |
+
|
| 101 |
+
NeMo:
|
| 102 |
+
|
| 103 |
+
```bash
|
| 104 |
+
NEMO_PYTHON=/opt/runtimes/nemo-speech-v3.0.0/.venv/bin/python \
|
| 105 |
+
NEMO_ALIGN_SCRIPT=/opt/runtimes/nemo-speech-v3.0.0/tools/nemo_forced_aligner/align.py \
|
| 106 |
+
NEMO_MODEL_PATH=/opt/models/stt_en_fastconformer_hybrid_large_pc.nemo \
|
| 107 |
+
NEMO_WAV_PATH=/absolute/path/to/file.wav \
|
| 108 |
+
NEMO_TRANSCRIPT='Exact authoritative transcript.' \
|
| 109 |
+
NEMO_DEVICE=cuda \
|
| 110 |
+
python -m pytest tests/integration/nemo/test_nemo_real.py -m 'integration and nemo' -v
|
| 111 |
+
```
|
| 112 |
+
|
| 113 |
+
NeMo may download the registry model only after `NEMO_ALLOW_MODEL_DOWNLOAD=1` is set
|
| 114 |
+
explicitly. A local checkpoint is preferred. Integration commands require the listed
|
| 115 |
+
hardware/data and are not part of ordinary unit CI.
|
| 116 |
+
|
| 117 |
+
## Implementation-to-Test and Coverage Map
|
| 118 |
+
|
| 119 |
+
Coverage values are current line coverage for each implementation file. Empty package
|
| 120 |
+
markers are intentionally zero bytes and excluded from implementation coverage; they
|
| 121 |
+
provide no executable behavior.
|
| 122 |
+
|
| 123 |
+
| Implementation | Focused test | Coverage |
|
| 124 |
+
|---|---|---:|
|
| 125 |
+
| `speech_text_data_aligners/__init__.py` | `tests/unit/test_public_api.py` | 100.00% |
|
| 126 |
+
| `application/annotate.py` | `tests/unit/application/test_annotate.py` | 98.00% |
|
| 127 |
+
| `application/benchmark.py` | `tests/unit/application/test_benchmark.py` | 100.00% |
|
| 128 |
+
| `application/materialize.py` | `tests/unit/application/test_materialize.py` | 95.68% |
|
| 129 |
+
| `application/readiness.py` | `tests/unit/application/test_readiness.py` | 100.00% |
|
| 130 |
+
| `backends/registry.py` | `tests/unit/backends/test_registry.py` | 100.00% |
|
| 131 |
+
| `backends/mfa/backend.py` | `tests/unit/backends/mfa/test_backend.py` | 100.00% |
|
| 132 |
+
| `backends/mfa/config.py` | `tests/unit/backends/mfa/test_config.py` | 99.03% |
|
| 133 |
+
| `backends/mfa/corpus.py` | `tests/unit/backends/mfa/test_corpus.py` | 99.19% |
|
| 134 |
+
| `backends/mfa/runner.py` | `tests/unit/backends/mfa/test_runner.py` | 98.62% |
|
| 135 |
+
| `backends/mfa/textgrid.py` | `tests/unit/backends/mfa/test_textgrid.py` | 97.91% |
|
| 136 |
+
| `backends/nemo/backend.py` | `tests/unit/backends/nemo/test_backend.py` | 100.00% |
|
| 137 |
+
| `backends/nemo/config.py` | `tests/unit/backends/nemo/test_config.py` | 98.11% |
|
| 138 |
+
| `backends/nemo/ctm.py` | `tests/unit/backends/nemo/test_ctm.py` | 100.00% |
|
| 139 |
+
| `backends/nemo/manifest.py` | `tests/unit/backends/nemo/test_manifest.py` | 100.00% |
|
| 140 |
+
| `backends/nemo/runner.py` | `tests/unit/backends/nemo/test_runner.py` | 100.00% |
|
| 141 |
+
| `backends/whisperx/backend.py` | `tests/unit/backends/whisperx/test_backend.py` | 100.00% |
|
| 142 |
+
| `backends/whisperx/config.py` | `tests/unit/backends/whisperx/test_config.py` | 100.00% |
|
| 143 |
+
| `backends/whisperx/parser.py` | `tests/unit/backends/whisperx/test_parser.py` | 100.00% |
|
| 144 |
+
| `backends/whisperx/provider.py` | `tests/unit/backends/whisperx/test_provider.py` | 98.74% |
|
| 145 |
+
| `backends/whisperx/segments.py` | `tests/unit/backends/whisperx/test_segments.py` | 100.00% |
|
| 146 |
+
| `backends/whisperx/worker.py` | `tests/unit/backends/whisperx/test_worker.py` | 93.33% |
|
| 147 |
+
| `benchmarking/metrics.py` | `tests/unit/benchmarking/test_metrics.py` | 100.00% |
|
| 148 |
+
| `benchmarking/readiness.py` | `tests/unit/benchmarking/test_readiness.py` | 100.00% |
|
| 149 |
+
| `benchmarking/reports.py` | `tests/unit/benchmarking/test_reports.py` | 98.84% |
|
| 150 |
+
| `cli/annotate.py` | `tests/unit/cli/test_annotate.py` | 100.00% |
|
| 151 |
+
| `cli/benchmark.py` | `tests/unit/cli/test_benchmark.py` | 100.00% |
|
| 152 |
+
| `cli/main.py` | `tests/unit/cli/test_main.py` | 95.83% |
|
| 153 |
+
| `cli/readiness.py` | `tests/unit/cli/test_readiness.py` | 100.00% |
|
| 154 |
+
| `core/errors.py` | `tests/unit/core/test_errors.py` | 100.00% |
|
| 155 |
+
| `core/extraction.py` | `tests/unit/core/test_extraction.py` | 95.65% |
|
| 156 |
+
| `core/models.py` | `tests/unit/core/test_models.py` | 100.00% |
|
| 157 |
+
| `core/normalization.py` | `tests/unit/core/test_normalization.py` | 100.00% |
|
| 158 |
+
| `core/protocols.py` | `tests/unit/core/test_protocols.py` | 100.00% |
|
| 159 |
+
| `core/readiness.py` | `tests/unit/core/test_readiness.py` | 100.00% |
|
| 160 |
+
| `core/validation.py` | `tests/unit/core/test_validation.py` | 98.33% |
|
| 161 |
+
| `data/audio.py` | `tests/unit/data/test_audio.py` | 97.37% |
|
| 162 |
+
| `data/metadata.py` | `tests/unit/data/test_metadata.py` | 98.08% |
|
| 163 |
+
| `data/readiness.py` | `tests/unit/data/test_readiness.py` | 100.00% |
|
| 164 |
+
| `runtime/alignment_evidence.py` | `tests/unit/runtime/test_alignment_evidence.py` | 100.00% |
|
| 165 |
+
| `runtime/artifacts.py` | `tests/unit/runtime/test_artifacts.py` | 100.00% |
|
| 166 |
+
| `runtime/process.py` | `tests/unit/runtime/test_process.py` | 100.00% |
|
| 167 |
+
| `runtime/receipts.py` | `tests/unit/runtime/test_receipts.py` | 100.00% |
|
| 168 |
+
|
| 169 |
+
## Coverage Interpretation
|
| 170 |
+
|
| 171 |
+
- Coverage is line coverage from the deterministic full suite with real integration
|
| 172 |
+
tests skipped.
|
| 173 |
+
- High coverage does not replace behavioral review or empirical alignment evidence.
|
| 174 |
+
- A skipped integration test is not counted as a successful real backend run.
|
| 175 |
+
- Aggregate coverage cannot compensate for a file below its individual threshold.
|
| 176 |
+
- When an implementation file is added, add its focused test and update this map in the
|
| 177 |
+
same change.
|
|
@@ -1,5 +1,9 @@
|
|
| 1 |
# W1-T semantic-readiness timing
|
| 2 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
W1-T maps an already frozen source-English semantic word boundary to an automatic
|
| 4 |
acoustic END estimate. It does not adjudicate the semantic boundary or measure human
|
| 5 |
Russian onset. Every sidecar record fixes `human_measured=false`,
|
|
|
|
| 1 |
# W1-T semantic-readiness timing
|
| 2 |
|
| 3 |
+
This document preserves the specialized W1-T workflow and its 2026-09-10 evidence
|
| 4 |
+
snapshot. For the current repository-wide quality command sequence and current test
|
| 5 |
+
counts, use [Testing and Quality](TESTING.md).
|
| 6 |
+
|
| 7 |
W1-T maps an already frozen source-English semantic word boundary to an automatic
|
| 8 |
acoustic END estimate. It does not adjudicate the semantic boundary or measure human
|
| 9 |
Russian onset. Every sidecar record fixes `human_measured=false`,
|
|
@@ -116,6 +116,39 @@ accuracy.
|
|
| 116 |
The Python benchmark service returns a typed report without writing. The CLI can print
|
| 117 |
one selected format or preflight and publish a deterministic JSON/CSV/Markdown bundle.
|
| 118 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 119 |
## NeMo Workflow
|
| 120 |
|
| 121 |
```text
|
|
@@ -259,3 +292,41 @@ required result/reference values exist.
|
|
| 259 |
For API examples and the concise entry point, return to the
|
| 260 |
[repository README](../README.md). For the current empirical boundary, see
|
| 261 |
[Project Status](PROJECT_STATUS.md).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 116 |
The Python benchmark service returns a typed report without writing. The CLI can print
|
| 117 |
one selected format or preflight and publish a deterministic JSON/CSV/Markdown bundle.
|
| 118 |
|
| 119 |
+
## Offline Readiness Workflow
|
| 120 |
+
|
| 121 |
+
```text
|
| 122 |
+
exact three-column metadata + frozen boundaries + exact WAV directory
|
| 123 |
+
|
|
| 124 |
+
v
|
| 125 |
+
validate cohort size, unique audio IDs, transcript/audio hashes, and boundaries
|
| 126 |
+
|
|
| 127 |
+
v
|
| 128 |
+
audit supplied full-word evidence capsules
|
| 129 |
+
+-- verified case/backend --> reuse
|
| 130 |
+
`-- missing/rejected ------> provider execution
|
| 131 |
+
|
|
| 132 |
+
+-- unavailable runtime --> structured status
|
| 133 |
+
+-- typed failure --------> structured status
|
| 134 |
+
`-- successful result ----> verify and save capsule
|
| 135 |
+
|
|
| 136 |
+
v
|
| 137 |
+
map frozen semantic token to original lexical span
|
| 138 |
+
|
|
| 139 |
+
v
|
| 140 |
+
extract mapped word END values and compute median/range/MAD
|
| 141 |
+
|
|
| 142 |
+
v
|
| 143 |
+
timing sidecar + summary + review queue + receipt + reuse ledger
|
| 144 |
+
```
|
| 145 |
+
|
| 146 |
+
The readiness CLI composes providers from an explicit catalog. It runs only missing or
|
| 147 |
+
rejected case/backend pairs and refuses to overwrite an existing output directory.
|
| 148 |
+
The service marks its sidecar `evaluation_only=true`, `human_measured=false`, and
|
| 149 |
+
`runtime_usable=false`. See [Configuration](CONFIGURATION.md) for catalog structure and
|
| 150 |
+
[W1-T Semantic Readiness](W1_T_SEMANTIC_READINESS.md) for mapping/consensus semantics.
|
| 151 |
+
|
| 152 |
## NeMo Workflow
|
| 153 |
|
| 154 |
```text
|
|
|
|
| 292 |
For API examples and the concise entry point, return to the
|
| 293 |
[repository README](../README.md). For the current empirical boundary, see
|
| 294 |
[Project Status](PROJECT_STATUS.md).
|
| 295 |
+
|
| 296 |
+
## Command and Artifact Map
|
| 297 |
+
|
| 298 |
+
| Command | Primary inputs | Primary outputs | Exit behavior |
|
| 299 |
+
|---|---|---|---|
|
| 300 |
+
| `annotate` | Metadata CSV, WAV directory, backend config | Annotated CSV; optional diagnostics/receipt | `0` success; typed project errors exit `2` |
|
| 301 |
+
| `benchmark` | Metadata, sparse reference, WAV directory, one or more backend configs | Rendered report or JSON/CSV/Markdown bundle | `0` after report; typed project errors exit `2` |
|
| 302 |
+
| `readiness` | Three-column metadata, frozen boundaries, WAV directory, catalog | Four primary W1-T JSON/JSONL artifacts plus reuse/attempt evidence | `0` timing-complete; `3` timing-blocked; typed errors exit `2` |
|
| 303 |
+
|
| 304 |
+
Annotation outputs:
|
| 305 |
+
|
| 306 |
+
| Artifact | When written | Contract |
|
| 307 |
+
|---|---|---|
|
| 308 |
+
| Annotated CSV | Every completed continue-on-error run and every successful strict run | Source preserved; exactly six columns appended |
|
| 309 |
+
| Diagnostics JSON | When one or more rows fail | Ordered structured failures; no fabricated timing |
|
| 310 |
+
| Receipt JSON | When requested | Input hash, backend, counts, host, timing, artifact paths |
|
| 311 |
+
|
| 312 |
+
Benchmark outputs:
|
| 313 |
+
|
| 314 |
+
| Artifact | Contract |
|
| 315 |
+
|---|---|
|
| 316 |
+
| JSON | Complete typed observations, gold metrics, pairwise metrics |
|
| 317 |
+
| CSV | Deterministic `scope,metric,value` headline rows |
|
| 318 |
+
| Markdown | Human-readable headline metrics |
|
| 319 |
+
|
| 320 |
+
Readiness outputs in a new attempt directory:
|
| 321 |
+
|
| 322 |
+
| Artifact | Contract |
|
| 323 |
+
|---|---|
|
| 324 |
+
| `W1_T_SEMANTIC_READINESS_TIMING.jsonl` | One ordered row per frozen case |
|
| 325 |
+
| `W1_T_SUMMARY.json` | Distribution, confidence, and acceptance summary |
|
| 326 |
+
| `W1_T_REVIEW_QUEUE.jsonl` | Failures, provenance gaps, ambiguity, and low confidence |
|
| 327 |
+
| `W1_T_RECEIPT.json` | Cohort hashes, host, timing gate, and status |
|
| 328 |
+
| `reuse_ledger*.jsonl` | Pre-run and final evidence reuse decisions |
|
| 329 |
+
| `backend_runs/` | Requests and new canonical result/failure evidence |
|
| 330 |
+
|
| 331 |
+
Configuration fields and exact JSON shapes are in [Configuration](CONFIGURATION.md).
|
| 332 |
+
Metric interpretation is in [Evaluation](EVALUATION.md).
|
|
@@ -0,0 +1,76 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# License: Apache-2.0 License
|
| 2 |
+
# Created by: Patrick Lumbantobing, VertoX-AI
|
| 3 |
+
# Copyright (c) 2026 VertoX-AI. All rights reserved.
|
| 4 |
+
#
|
| 5 |
+
# This work is licensed under the Apache-2.0 License.
|
| 6 |
+
# To view a copy of this license, visit
|
| 7 |
+
# https://www.apache.org/licenses/LICENSE-2.0
|
| 8 |
+
|
| 9 |
+
"""Validate the machine-readable project manifest against stable source contracts."""
|
| 10 |
+
|
| 11 |
+
from __future__ import annotations
|
| 12 |
+
|
| 13 |
+
import hashlib
|
| 14 |
+
import json
|
| 15 |
+
import tomllib
|
| 16 |
+
from pathlib import Path
|
| 17 |
+
|
| 18 |
+
from speech_text_data_aligners.backends.mfa.config import (
|
| 19 |
+
DEFAULT_MODEL_ID as MFA_MODEL_ID,
|
| 20 |
+
)
|
| 21 |
+
from speech_text_data_aligners.backends.mfa.config import (
|
| 22 |
+
DEFAULT_MODEL_REVISION as MFA_REVISION,
|
| 23 |
+
)
|
| 24 |
+
from speech_text_data_aligners.backends.nemo.config import (
|
| 25 |
+
DEFAULT_MODEL_ID as NEMO_MODEL_ID,
|
| 26 |
+
)
|
| 27 |
+
from speech_text_data_aligners.backends.registry import default_backend_registry
|
| 28 |
+
from speech_text_data_aligners.backends.whisperx.config import (
|
| 29 |
+
DEFAULT_MODEL_ID as WHISPERX_MODEL_ID,
|
| 30 |
+
)
|
| 31 |
+
from speech_text_data_aligners.core.extraction import BOUNDARY_COLUMNS
|
| 32 |
+
from speech_text_data_aligners.data.metadata import SOURCE_COLUMNS
|
| 33 |
+
|
| 34 |
+
|
| 35 |
+
def _manifest() -> dict[str, object]:
|
| 36 |
+
"""Load the root manifest as a JSON object.
|
| 37 |
+
|
| 38 |
+
Returns
|
| 39 |
+
-------
|
| 40 |
+
Parsed manifest values.
|
| 41 |
+
|
| 42 |
+
"""
|
| 43 |
+
return json.loads(Path("config.json").read_text(encoding="utf-8"))
|
| 44 |
+
|
| 45 |
+
|
| 46 |
+
def test_manifest_matches_registry_schema_and_canonical_values() -> None:
|
| 47 |
+
"""Assert that the manifest cannot drift from registry/schema/model defaults."""
|
| 48 |
+
manifest = _manifest()
|
| 49 |
+
assert manifest["schema_version"] == "speech-text-data-aligners/config.v1"
|
| 50 |
+
data_contract = manifest["data_contract"]
|
| 51 |
+
assert data_contract["source_columns"] == list(SOURCE_COLUMNS)
|
| 52 |
+
assert data_contract["alignment_columns"] == list(BOUNDARY_COLUMNS)
|
| 53 |
+
registry = manifest["backend_registry"]
|
| 54 |
+
assert registry["operational_selection"] == "mfa"
|
| 55 |
+
assert set(registry["models"]) == set(default_backend_registry().names)
|
| 56 |
+
models = registry["models"]
|
| 57 |
+
assert models["mfa"]["model"]["id"] == MFA_MODEL_ID
|
| 58 |
+
assert models["mfa"]["model"]["revision"] == MFA_REVISION
|
| 59 |
+
assert models["nemo"]["model"]["id"] == NEMO_MODEL_ID
|
| 60 |
+
assert models["whisperx"]["model"]["id"] == WHISPERX_MODEL_ID
|
| 61 |
+
evaluation = manifest["evaluation"]
|
| 62 |
+
assert evaluation["manual_gold"]["rows"] == 5
|
| 63 |
+
assert evaluation["manual_gold"]["fields"] == [
|
| 64 |
+
"first_word_text",
|
| 65 |
+
"first_word_end_s",
|
| 66 |
+
]
|
| 67 |
+
assert (
|
| 68 |
+
evaluation["canonical_output"]["path"] == "results/final/final-annotated-36.csv"
|
| 69 |
+
)
|
| 70 |
+
project_config = tomllib.loads(Path("pyproject.toml").read_text(encoding="utf-8"))
|
| 71 |
+
assert manifest["project"]["version"] == project_config["project"]["version"]
|
| 72 |
+
canonical = Path(evaluation["canonical_output"]["path"]).read_bytes()
|
| 73 |
+
assert (
|
| 74 |
+
evaluation["canonical_output"]["sha256"]
|
| 75 |
+
== hashlib.sha256(canonical).hexdigest()
|
| 76 |
+
)
|
|
@@ -0,0 +1,67 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# License: Apache-2.0 License
|
| 2 |
+
# Created by: Patrick Lumbantobing, VertoX-AI
|
| 3 |
+
# Copyright (c) 2026 VertoX-AI. All rights reserved.
|
| 4 |
+
#
|
| 5 |
+
# This work is licensed under the Apache-2.0 License.
|
| 6 |
+
# To view a copy of this license, visit
|
| 7 |
+
# https://www.apache.org/licenses/LICENSE-2.0
|
| 8 |
+
|
| 9 |
+
"""Verify that repository Markdown documents resolve their local link targets."""
|
| 10 |
+
|
| 11 |
+
from __future__ import annotations
|
| 12 |
+
|
| 13 |
+
import re
|
| 14 |
+
from pathlib import Path
|
| 15 |
+
from urllib.parse import unquote, urlsplit
|
| 16 |
+
|
| 17 |
+
PROJECT_ROOT = Path(__file__).resolve().parents[2]
|
| 18 |
+
MARKDOWN_LINK = re.compile(r"(?<!\!)\[[^\]]*\]\(([^)\s]+)(?:\s+[^)]*)?\)")
|
| 19 |
+
SKIP_PREFIXES = ("http://", "https://", "mailto:", "data:", "#")
|
| 20 |
+
|
| 21 |
+
|
| 22 |
+
def _markdown_files() -> tuple[Path, ...]:
|
| 23 |
+
"""Return maintained repository Markdown files in deterministic order.
|
| 24 |
+
|
| 25 |
+
Returns
|
| 26 |
+
-------
|
| 27 |
+
Root and tracked-area Markdown paths excluding generated/agent-private trees.
|
| 28 |
+
|
| 29 |
+
"""
|
| 30 |
+
return tuple(
|
| 31 |
+
sorted(
|
| 32 |
+
path
|
| 33 |
+
for path in PROJECT_ROOT.rglob("*.md")
|
| 34 |
+
if not any(
|
| 35 |
+
part
|
| 36 |
+
in {".agents", ".git", ".mypy_cache", ".pytest_cache", ".ruff_cache"}
|
| 37 |
+
for part in path.relative_to(PROJECT_ROOT).parts
|
| 38 |
+
)
|
| 39 |
+
)
|
| 40 |
+
)
|
| 41 |
+
|
| 42 |
+
|
| 43 |
+
def test_local_markdown_links_resolve() -> None:
|
| 44 |
+
"""Fail when any local Markdown link targets a missing path."""
|
| 45 |
+
failures: list[str] = []
|
| 46 |
+
for document in _markdown_files():
|
| 47 |
+
content = document.read_text(encoding="utf-8")
|
| 48 |
+
for raw_target in MARKDOWN_LINK.findall(content):
|
| 49 |
+
target = unquote(urlsplit(raw_target).path)
|
| 50 |
+
if not target or raw_target.startswith(SKIP_PREFIXES):
|
| 51 |
+
continue
|
| 52 |
+
resolved = (
|
| 53 |
+
Path(target) if target.startswith("/") else document.parent / target
|
| 54 |
+
).resolve()
|
| 55 |
+
try:
|
| 56 |
+
resolved.relative_to(PROJECT_ROOT.resolve())
|
| 57 |
+
except ValueError:
|
| 58 |
+
failures.append(
|
| 59 |
+
f"{document.relative_to(PROJECT_ROOT)}: link escapes repository: "
|
| 60 |
+
f"{raw_target}"
|
| 61 |
+
)
|
| 62 |
+
continue
|
| 63 |
+
if not resolved.exists():
|
| 64 |
+
failures.append(
|
| 65 |
+
f"{document.relative_to(PROJECT_ROOT)}: missing target: {raw_target}"
|
| 66 |
+
)
|
| 67 |
+
assert not failures, "broken local Markdown links:\n" + "\n".join(failures)
|