pltobing commited on
Commit
a3b146e
·
1 Parent(s): d6e2489

docs: refine aligner documentation

Browse files

- Rebuild README.md as a Hugging Face-style framework card with tags, license, model/configuration coverage, usage, metrics, limitations, and references.
- Add config.json as a tested machine-readable project/model manifest and bind its schema, versions, backends, model identities, and canonical hash to source/evidence values.
- Add root ARCHITECTURE.md and focused configuration, evaluation, testing, workflow, and documentation-map guidance for current operating use.
- Preserve historical reports while clarifying current contracts, empirical boundaries, readiness outputs, and the selected MFA operational configuration.
- Add deterministic manifest and local Markdown-link quality tests with approved Python headers.
- Record plans, review drafts, baseline/final work data, and validation logs under .agents while excluding private and raw orchestrator artifacts.
- Update AGENTS.md and SKILLS.md with durable documentation and manifest integrity requirements.

.agents/drafts/documentation-refinement-review.md ADDED
@@ -0,0 +1,67 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Documentation Refinement Review Draft
2
+
3
+ Date: 2026-10-30
4
+
5
+ ## Purpose
6
+
7
+ Turn the repository's implementation, contracts, configuration, and empirical evidence
8
+ into a concise documentation set that is accurate enough to use without reading the
9
+ historical stage reports.
10
+
11
+ ## Baseline findings
12
+
13
+ - The library-first architecture and authoritative transcript contract are implemented
14
+ and enforced by deterministic tests and seven import-linter contracts.
15
+ - The README has useful model-card elements but lacks complete tags, an explicit
16
+ configuration matrix, metric definitions, evaluation-table caveats, and a clear
17
+ distinction between current guidance and archival reports.
18
+ - `docs/ARCHITECTURE.md` omits the current readiness, evidence, and materialization
19
+ modules from its package tree and layer narrative.
20
+ - `docs/WORKFLOWS.md` covers annotation and benchmarking well but does not provide one
21
+ authoritative CLI/API configuration reference or report artifact contract.
22
+ - Backend defaults, model identifiers, immutable revisions, runtime versions, artifact
23
+ policies, and required path fields are spread across source docstrings and reports.
24
+ - No root machine-readable configuration manifest currently complements the model-card
25
+ README; the requested `config.json` should record canonical project/model metadata and
26
+ pinned backend/model identities without pretending to be a Transformers checkpoint.
27
+ - Evaluation numbers are accurate but need one canonical comparison page that states
28
+ sample size, supported gold fields, aggregation rules, runtime caveats, and the
29
+ boundary between accuracy and pairwise disagreement.
30
+ - Existing stage reports are valuable provenance but should be labeled as historical
31
+ evidence rather than current operating instructions.
32
+ - AGENTS.md and SKILLS.md are accurate for implementation work. The documentation task
33
+ should add review/verification workflow guidance without weakening existing gates.
34
+
35
+ ## Planned documentation changes
36
+
37
+ 1. Record the review baseline and per-file coverage evidence under `.agents/work_data/`.
38
+ 2. Add `config.json` as a versioned machine-readable project/model manifest.
39
+ 3. Rewrite README.md as the Hugging Face-style entry point with model metadata, tags,
40
+ supported models/configurations, quick starts, metrics, limitations, and references.
41
+ 4. Expand the architecture documentation to include readiness/evidence/materialization
42
+ responsibilities and current dependency boundaries.
43
+ 5. Add focused configuration, evaluation, and testing/quality references.
44
+ 6. Refine workflows and project status so commands, outputs, exit behavior, and evidence
45
+ paths are easy to locate.
46
+ 7. Add a documentation map that separates current contracts from historical reports.
47
+ 8. Validate links, CLI help, deterministic tests, lint/type/import gates, coverage, and
48
+ Python header policy; record results in `.agents/logs/`.
49
+ 9. Review AGENTS.md and SKILLS.md again and make only necessary documentation-quality
50
+ updates.
51
+
52
+ ## Non-goals
53
+
54
+ - Do not change source metadata, translations, canonical result CSVs, or empirical
55
+ evidence artifacts.
56
+ - Do not claim manual accuracy for unannotated boundaries.
57
+ - Do not add implementation scripts solely for documentation generation.
58
+ - Do not push to a remote repository.
59
+
60
+ ## Finalized outcome
61
+
62
+ - Added a Hugging Face-style README and machine-readable `config.json` manifest.
63
+ - Added focused architecture, configuration, evaluation, workflow, testing, and
64
+ documentation-map guides.
65
+ - Added deterministic manifest and local-link quality tests.
66
+ - Updated AGENTS.md/SKILLS.md with documentation and manifest integrity guidance.
67
+ - Preserved historical evidence and source metadata unchanged.
.agents/logs/documentation-refinement-20261030T152745Z.md ADDED
@@ -0,0 +1,43 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Documentation Refinement Log
2
+
3
+ ## 2026-10-30T15:27:45Z - Start
4
+
5
+ - Read AGENTS.md, SKILLS.md, the aligner engineering contract, and the modular
6
+ architecture contract before implementation.
7
+ - Inventoried tracked source, tests, environments, documentation, results, and untracked
8
+ `.agents/` materials.
9
+ - Compared README and current docs with public exports, CLI help, typed backend configs,
10
+ application services, metrics/report schemas, readiness flow, and environment specs.
11
+ - Recorded baseline deterministic evidence: 633 passed / 3 skipped, 98.74% aggregate
12
+ coverage, lowest file 93.33%, Ruff pass, strict mypy pass, 7 import contracts kept.
13
+ - Created the requested review draft, execution plan, and baseline work data before
14
+ changing user-facing documentation.
15
+
16
+ ## 2026-10-30 - Documentation implementation
17
+
18
+ - Preserved the user-authored `.gitignore` exclusions for `.agents/config`,
19
+ `.agents/user_prompts`, and `.agents/user_scripts`.
20
+ - Added root `config.json` as a versioned project/model manifest and tested its schema,
21
+ registry/model identity, package version, and canonical result hash.
22
+ - Reworked README.md into a Hugging Face-style model/framework card with tags, license,
23
+ model/configuration comparison, usage, metrics, limitations, and references.
24
+ - Added root ARCHITECTURE.md plus focused CONFIGURATION.md, EVALUATION.md, TESTING.md,
25
+ and DOCUMENTATION.md guides; refined WORKFLOWS.md, PROJECT_STATUS.md, the architecture
26
+ contract, and W1-T provenance labeling.
27
+ - Added deterministic local Markdown-link validation and finalized AGENTS.md/SKILLS.md
28
+ documentation-integrity guidance.
29
+
30
+ ## 2026-10-30 - Final verification
31
+
32
+ - `python -m pytest -q`: 635 passed, 3 skipped.
33
+ - `coverage run -m pytest -q`: 635 passed, 3 skipped.
34
+ - `coverage report --format=total`: 98.74%; lowest implementation file 93.33%.
35
+ - `python -m ruff check .`: pass.
36
+ - `python -m ruff format --check .`: 150 files already formatted.
37
+ - `python -m mypy src`: pass across 53 source files.
38
+ - `lint-imports`: 7 contracts kept, 0 broken.
39
+ - Architecture tests: 10 passed.
40
+ - Header, manifest, documentation-link, and per-file coverage tests: pass.
41
+ - `speech-text-aligners` and all three subcommand help surfaces: pass.
42
+ - `conda env create --help`, `python -m pip install --help`, and the named Conda
43
+ environment Python check: pass.
.agents/plans/01-review-and-documentation-refinement.md ADDED
@@ -0,0 +1,63 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Plan: Repository Review and Documentation Refinement
2
+
3
+ Status: complete
4
+
5
+ ## Objective
6
+
7
+ Produce a comprehensive, navigable, and reproducible documentation set for using,
8
+ configuring, extending, evaluating, and verifying Speech-Text Data Aligners.
9
+
10
+ ## Execution
11
+
12
+ ### Review repository contracts and implementation
13
+
14
+ - Read AGENTS.md, SKILLS.md, both mandatory contracts, public models/protocols, application
15
+ services, CLI composition, backend configurations, environment specs, and evidence
16
+ summaries.
17
+ - Compare documented commands and APIs with parser options and source signatures.
18
+ - Identify current, historical, and evidence-only documents.
19
+
20
+ ### Draft the documentation architecture
21
+
22
+ - Store gap analysis and baseline evidence in `.agents/drafts/` and `.agents/work_data/`.
23
+ - Define README as the model-card and quick-start entry point.
24
+ - Define focused documents for architecture, configuration, evaluation, workflows,
25
+ testing/quality, and project status.
26
+ - Keep immutable historical reports unchanged except for navigational corrections.
27
+
28
+ ### Refine public documentation
29
+
30
+ - Add `config.json` as a versioned machine-readable repository/model manifest.
31
+ - Expand Hugging Face-style metadata and tags without inventing unavailable model fields.
32
+ - Document all three alignment models/backends and every material configuration family.
33
+ - Document Python and CLI usage for annotation, benchmarking, and offline readiness.
34
+ - Define schema fields, timestamp semantics, failure policies, output artifacts, and
35
+ provenance requirements.
36
+ - Publish one performance comparison with explicit gold coverage and non-accuracy
37
+ caveats.
38
+ - Make current versus archival evidence and canonical artifact paths explicit.
39
+
40
+ ### Verify acceptance gates
41
+
42
+ - Check local Markdown links and referenced repository paths.
43
+ - Run CLI help for every command.
44
+ - Run Ruff, mypy, import-linter, deterministic tests, architecture tests, header tests,
45
+ fresh coverage, and per-file coverage enforcement.
46
+ - Record observed commands/results in `.agents/logs/`.
47
+ - Review AGENTS.md and SKILLS.md at completion and update only if a durable gap is found.
48
+
49
+ ## Acceptance criteria
50
+
51
+ - A new user can install an environment and run each documented entry point without
52
+ consulting stage reports.
53
+ - Every documented command has passed at least syntax/help-level verification.
54
+ - Configuration tables match typed config fields and defaults in source.
55
+ - `config.json` is valid JSON and its canonical values agree with `pyproject.toml`, the
56
+ engineering contract, and pinned backend configuration defaults.
57
+ - Metrics are tied to exact sample sizes and reference fields.
58
+ - Architecture documentation covers all current layers and modules.
59
+ - Tests remain at or above the repository gate: >=90% per implementation file and >=90%
60
+ aggregate coverage.
61
+ - Documentation-only work leaves source data/results unchanged.
62
+ - Work is committed on a branch and merged to main without a remote push.
63
+ - Final git status is clean.
.agents/work_data/documentation-final-verification.md ADDED
@@ -0,0 +1,24 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Documentation Final Verification
2
+
3
+ Date: 2026-10-30
4
+
5
+ | Check | Result |
6
+ |---|---|
7
+ | Full deterministic tests | 635 passed, 3 skipped |
8
+ | Coverage run | 635 passed, 3 skipped |
9
+ | Aggregate implementation line coverage | 98.74% |
10
+ | Lowest implementation-file coverage | 93.33% (`backends/whisperx/worker.py`) |
11
+ | Ruff lint | Pass |
12
+ | Ruff format | 150 files already formatted |
13
+ | Strict mypy | Pass across 53 source files |
14
+ | Import Linter | 7 kept, 0 broken |
15
+ | Architecture tests | 10 passed |
16
+ | Header/manifest/link/per-file tests | Pass |
17
+ | CLI top-level and subcommand help | Pass |
18
+ | Conda/pip setup command help | Pass |
19
+ | Named `speech-text-data-aligners` environment | Python 3.12.15 available |
20
+ | Local Markdown links | All maintained targets resolve |
21
+ | Source metadata/results | Unchanged |
22
+
23
+ Coverage remains within the repository gate for every non-empty implementation file.
24
+ Heavyweight integration tests remain opt-in and were skipped in ordinary CI.
.agents/work_data/documentation-review-baseline.md ADDED
@@ -0,0 +1,72 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Documentation Review Baseline
2
+
3
+ Captured: 2026-10-30T15:27:45Z
4
+
5
+ ## Repository state
6
+
7
+ - Base branch: `main`
8
+ - Review branch: `docs/documentation-refinement`
9
+ - Tracked implementation modules: 43 non-empty package Python files plus 10 empty
10
+ package markers
11
+ - Test modules mirror core, data, application, benchmarking, runtime, backend, CLI,
12
+ architecture, quality, and opt-in integration boundaries
13
+
14
+ ## Observed deterministic quality results
15
+
16
+ | Gate | Result |
17
+ |---|---|
18
+ | Pytest | 633 passed, 3 skipped in 0.90 s |
19
+ | Coverage run | 633 passed, 3 skipped in 1.21 s |
20
+ | Aggregate line coverage | 98.74% |
21
+ | Lowest implementation-file coverage | 93.33% (`backends/whisperx/worker.py`) |
22
+ | Shared-layer minimum | 95.65% (`core/extraction.py`) |
23
+ | Ruff | All checks passed |
24
+ | mypy strict | Success: no issues found in 53 source files |
25
+ | Import Linter | 7 contracts kept, 0 broken |
26
+ | Architecture tests | 10 passed |
27
+ | Header/per-file quality tests | 4 passed |
28
+
29
+ ## Current documentation inventory
30
+
31
+ Current guidance:
32
+
33
+ - `README.md`
34
+ - `docs/ARCHITECTURE.md`
35
+ - `docs/WORKFLOWS.md`
36
+ - `docs/PROJECT_STATUS.md`
37
+ - `docs/ALIGNER_ENGINEERING_CONTRACT.md`
38
+ - `docs/MODULAR_ARCHITECTURE_CONTRACT.md`
39
+ - `docs/W1_T_SEMANTIC_READINESS.md`
40
+
41
+ Decision and empirical evidence:
42
+
43
+ - `docs/FINAL_ALIGNER_SELECTION.md`
44
+ - `docs/POSTFINAL_THREE_ALIGNER_COMPARISON.md`
45
+ - `results/final/`
46
+ - `results/postfinal/`
47
+ - `results/w1t/`
48
+
49
+ Historical acceptance/provenance:
50
+
51
+ - `docs/STAGE_0_REPORT.md`
52
+ - `docs/STAGE_1_REPORT.md`
53
+ - `docs/STAGE_2_REPORT.md`
54
+ - `docs/STAGE_3_REPORT.md`
55
+ - `docs/STAGE_4_REPORT.md`
56
+ - `docs/STAGE_5_FINAL_ACCEPTANCE_REPORT.md`
57
+ - `docs/POST_STAGE5_DOCS_LICENSE_REPORT.md`
58
+
59
+ ## Confirmed implementation facts for documentation
60
+
61
+ - Input columns are `line_id,audio_id,en_transcript,ru_translation`.
62
+ - Audio resolves strictly as `<audio_dir>/<audio_id>.wav`.
63
+ - Output appends six first/last lexical boundary fields and preserves source cells/order.
64
+ - CLI commands are `annotate`, `benchmark`, and `readiness`.
65
+ - Annotation accepts one backend and a backend-specific JSON object.
66
+ - Benchmark accepts repeated backend names and a JSON object keyed by backend name.
67
+ - Readiness accepts a provider/evidence catalog and writes a new output directory.
68
+ - Root public API exports annotation, benchmarking, neutral models, and `AlignmentBackend`.
69
+ - Specialized readiness/materialization services remain directly importable from their
70
+ application modules.
71
+ - Manual gold is limited to `first_word_text` and `first_word_end_s` on five rows.
72
+ - Pairwise timing differences are disagreement diagnostics, not accuracy estimates.
.gitignore CHANGED
@@ -8,3 +8,10 @@ coverage.json
8
  *.egg-info/
9
  dist/
10
  build/
 
 
 
 
 
 
 
 
8
  *.egg-info/
9
  dist/
10
  build/
11
+ .agents/config
12
+ .agents/user_prompts
13
+ .agents/user_scripts
14
+ .agents/logs/orchestrator-*.log
15
+ .*.swp
16
+ .*.swo
17
+ .agents/logs/codex-*.md
AGENTS.md CHANGED
@@ -113,6 +113,13 @@ For each empirical run record at least:
113
 
114
  Keep README and `docs/` aligned with actual commands and public Python APIs. Do not document commands that have not at least passed syntax/help-level checks, and clearly label commands that require unavailable hardware/data.
115
 
 
 
 
 
 
 
 
116
  ## Scope discipline
117
 
118
  Do not refactor unrelated code. Do not change source transcripts or Russian translations. Do not overwrite source CSVs by default. Do not commit generated model weights, caches, large alignment outputs, WAV data, or credentials.
 
113
 
114
  Keep README and `docs/` aligned with actual commands and public Python APIs. Do not document commands that have not at least passed syntax/help-level checks, and clearly label commands that require unavailable hardware/data.
115
 
116
+ Maintain `config.json` as a valid machine-readable project/model manifest, not as a
117
+ Transformers checkpoint configuration. Keep its schema, package version, registered
118
+ backends, model identities, data contract, evaluation scope, and canonical output hash
119
+ consistent with source and evidence through deterministic tests. Keep a documentation
120
+ map that distinguishes current guidance, contracts, empirical evidence, and historical
121
+ acceptance reports. Validate local Markdown links before completion.
122
+
123
  ## Scope discipline
124
 
125
  Do not refactor unrelated code. Do not change source transcripts or Russian translations. Do not overwrite source CSVs by default. Do not commit generated model weights, caches, large alignment outputs, WAV data, or credentials.
ARCHITECTURE.md ADDED
@@ -0,0 +1,259 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Architecture
2
+
3
+ Speech-Text Data Aligners is a library-first forced-alignment framework. The core
4
+ contract accepts WAV audio plus a known, authoritative transcript and returns ordered
5
+ backend-neutral timed words. The CLI is a composition and presentation layer.
6
+
7
+ The mandatory design rules are in
8
+ [`docs/MODULAR_ARCHITECTURE_CONTRACT.md`](docs/MODULAR_ARCHITECTURE_CONTRACT.md).
9
+ Runtime alignment semantics are in
10
+ [`docs/ALIGNER_ENGINEERING_CONTRACT.md`](docs/ALIGNER_ENGINEERING_CONTRACT.md).
11
+
12
+ ## Design Goals
13
+
14
+ - Keep authoritative reference text immutable.
15
+ - Keep core logic deterministic and independent of model/runtime packages.
16
+ - Make MFA, NeMo, and WhisperX interchangeable through `AlignmentBackend`.
17
+ - Contain native formats inside their adapters.
18
+ - Provide reusable Python services rather than CLI-only workflows.
19
+ - Keep optional runtimes independently unavailable.
20
+ - Preserve source metadata and deterministic output order.
21
+ - Represent failure explicitly and never fabricate timing.
22
+
23
+ ## Layer and Dependency Direction
24
+
25
+ ```text
26
+ CLI --------------------> Application --------------------> Core
27
+ | ^
28
+ +----------> Data -------------+
29
+ +----------> Benchmarking -----+
30
+ +----------> Runtime ----------+
31
+
32
+ CLI -- lazy composition --> Concrete Backends
33
+ | |
34
+ +------------> Core
35
+ +------------> Runtime
36
+ ```
37
+
38
+ Enforced consequences:
39
+
40
+ - Core does not import application, data, benchmarking, runtime, CLI, or backends.
41
+ - Data and benchmarking do not import concrete backends or CLI.
42
+ - Application services receive `AlignmentBackend` implementations by injection and do
43
+ not branch on concrete backend names.
44
+ - Concrete backend packages may depend on core/runtime and their own native modules,
45
+ but never on a sibling backend.
46
+ - CLI composition may select a registry factory; it does not own alignment logic.
47
+ - Backend factories are lazy dotted references, so importing the public library does
48
+ not load heavyweight aligner dependencies.
49
+
50
+ ## Current Package Tree
51
+
52
+ ```text
53
+ src/speech_text_data_aligners/
54
+ |-- __init__.py stable public facade
55
+ |-- core/
56
+ | |-- models.py immutable neutral request/result/provenance values
57
+ | |-- protocols.py AlignmentBackend and structural contracts
58
+ | |-- errors.py project exception/failure taxonomy
59
+ | |-- normalization.py lexical plans and monotonic reconciliation
60
+ | |-- validation.py timing, lexical, and publication-path invariants
61
+ | |-- extraction.py first/last word output fields
62
+ | `-- readiness.py semantic-boundary mapping and consensus
63
+ |-- data/
64
+ | |-- metadata.py preserving, atomic CSV input/output
65
+ | |-- audio.py audio resolution and lightweight WAV probing
66
+ | `-- readiness.py frozen readiness cohort/hash verification
67
+ |-- application/
68
+ | |-- annotate.py reusable annotation orchestration
69
+ | |-- benchmark.py identical-request benchmark orchestration
70
+ | |-- readiness.py evidence reuse and offline readiness execution
71
+ | `-- materialize.py accepted-evidence full-word timing materialization
72
+ |-- benchmarking/
73
+ | |-- metrics.py sparse-gold, runtime, and pairwise metrics
74
+ | |-- reports.py deterministic JSON/CSV/Markdown reporting
75
+ | `-- readiness.py readiness distributions and sensitivity summaries
76
+ |-- runtime/
77
+ | |-- process.py shell-free injectable subprocess boundary
78
+ | |-- artifacts.py temporary/retained artifact lifecycle
79
+ | |-- receipts.py hashes, host facts, and run receipts
80
+ | `-- alignment_evidence.py full-word evidence integrity and verification
81
+ |-- backends/
82
+ | |-- registry.py immutable lazy backend factories
83
+ | |-- nemo/
84
+ | | |-- config.py typed NFA runtime/model configuration
85
+ | | |-- manifest.py reference/output manifest parsing
86
+ | | |-- runner.py isolated NFA execution
87
+ | | |-- ctm.py CTM parsing/mapping
88
+ | | `-- backend.py neutral NFA adapter
89
+ | |-- mfa/
90
+ | | |-- config.py typed MFA runtime/model configuration
91
+ | | |-- corpus.py temporary corpus/label construction
92
+ | | |-- runner.py isolated MFA execution
93
+ | | |-- textgrid.py TextGrid parsing/mapping
94
+ | | `-- backend.py neutral MFA adapter
95
+ | `-- whisperx/
96
+ | |-- config.py typed worker/runtime/model configuration
97
+ | |-- segments.py full-duration reference segments
98
+ | |-- provider.py isolated-worker transport
99
+ | |-- worker.py lazy heavyweight native runtime
100
+ | |-- parser.py native word parsing/mapping
101
+ | `-- backend.py neutral WhisperX adapter
102
+ `-- cli/
103
+ |-- main.py command dispatch and typed-error exit status
104
+ |-- annotate.py annotation argument adapter
105
+ |-- benchmark.py benchmark argument adapter
106
+ `-- readiness.py readiness catalog/composition adapter
107
+ ```
108
+
109
+ Empty package-marker `__init__.py` files are deliberately zero bytes. Each non-empty
110
+ implementation module has a focused test module and meets its individual line-coverage
111
+ gate.
112
+
113
+ ## Module Responsibilities
114
+
115
+ | Area | Owns | Does not own |
116
+ |---|---|---|
117
+ | `core` | Neutral values, normalization/reconciliation, validation, extraction, readiness rules | CSV, subprocesses, native formats, model loading |
118
+ | `data` | Metadata/audio/cohort input and source preservation | Backend selection or metrics |
119
+ | `application` | Reuse-case orchestration and publication policy | Native parsing or benchmark math |
120
+ | `benchmarking` | Deterministic metrics and report rendering | Backend execution or native result dictionaries |
121
+ | `runtime` | Process, artifact, receipt, and evidence integrity primitives | Application row loops or model semantics |
122
+ | `backends.*` | Config, native inputs/outputs, execution, parsing, mapping | Metadata orchestration or sibling backend imports |
123
+ | `cli` | Argument parsing, backend composition, presentation | Alignment, CSV loops, metric computation |
124
+
125
+ ## Shared Alignment Contract
126
+
127
+ ```python
128
+ result = backend.align(request)
129
+ outcomes = backend.align_batch(requests)
130
+ ```
131
+
132
+ `AlignmentRequest` contains:
133
+
134
+ - caller/request identity;
135
+ - resolved WAV path;
136
+ - exact authoritative transcript;
137
+ - probed WAV duration;
138
+ - immutable JSON-safe context.
139
+
140
+ A successful `AlignmentResult` contains one `WordAlignment` per original lexical word,
141
+ plus provenance, diagnostics, and measured runtime. A failure is represented by
142
+ `AlignmentFailure`; it contains no timing fields. `AlignmentOutcome` enforces exactly
143
+ one result/failure variant.
144
+
145
+ The root package exports `AlignmentBackend`, neutral request/result/failure/provenance
146
+ models, annotation services, and benchmark services. Custom backends can implement the
147
+ protocol without importing a concrete adapter.
148
+
149
+ ## Transcript and Alignment Flow
150
+
151
+ ```text
152
+ source metadata
153
+ |
154
+ +-- immutable en_transcript
155
+ |
156
+ +-- resolve <audio_dir>/<audio_id>.wav and probe duration
157
+ v
158
+ lexical preflight and original-character mapping
159
+ v
160
+ backend-specific reversible reference representation
161
+ v
162
+ native model/process execution
163
+ v
164
+ backend-local native output parsing
165
+ v
166
+ shared monotonic reconciliation and validation
167
+ v
168
+ WordAlignment / AlignmentFailure
169
+ v
170
+ first/last extraction or benchmark observation
171
+ ```
172
+
173
+ Backends may normalize punctuation, case, numbers, abbreviations, or native token
174
+ units internally. Every successful native token must map deterministically to the
175
+ original transcript's lexical words. Shared validation rejects incomplete coverage,
176
+ non-finite/negative/reversed/overlapping timing, and out-of-audio values. It never
177
+ clamps, interpolates, extrapolates, or invents a timestamp.
178
+
179
+ ## Native Boundary Containment
180
+
181
+ | Backend | Native structures contained locally | External boundary |
182
+ |---|---|---|
183
+ | NeMo | Reference/output manifests and word CTM rows | Isolated NFA process |
184
+ | MFA | Temporary WAV/LAB corpus and TextGrid tiers | Isolated Conda MFA process |
185
+ | WhisperX | Reference segments and word-segment dictionaries | Isolated heavyweight worker |
186
+
187
+ Only neutral models, diagnostics, and provenance cross these boundaries. Parent-process
188
+ imports do not require NeMo, MFA, WhisperX, Torch, or Torchaudio.
189
+
190
+ ## Annotation Publication
191
+
192
+ `application.annotate` prepares all source rows, invokes one injected backend, validates
193
+ and merges outcomes in source order, extracts six boundary columns, and atomically
194
+ publishes the enriched CSV. Continue-on-error preserves failed source rows with blank
195
+ alignment cells and structured diagnostics. Strict mode publishes no annotated output
196
+ after a row failure.
197
+
198
+ ## Benchmark Composition
199
+
200
+ `application.benchmark` constructs one immutable ordered request tuple and supplies it
201
+ to every injected backend. It joins sparse manual gold by `(line_id, audio_id)`, scores
202
+ available first-word text/end fields, and compares successful first/last boundaries
203
+ pairwise. Reports are backend-neutral; native artifact paths appear only as opaque
204
+ references.
205
+
206
+ ## Offline Readiness and Evidence
207
+
208
+ The readiness extension retains the same backend isolation:
209
+
210
+ ```text
211
+ metadata + frozen boundaries + WAVs
212
+ |
213
+ v
214
+ cohort/hash verification
215
+ |
216
+ +-- verified full-word evidence capsules --> reuse audit
217
+ |
218
+ `-- missing/rejected case-backend pairs --> injected AlignmentBackend
219
+ |
220
+ v
221
+ full-word evidence verification
222
+ |
223
+ v
224
+ boundary mapping + consensus/reporting
225
+ ```
226
+
227
+ `core.readiness` owns mapping/extraction/consensus, `data.readiness` owns cohort
228
+ verification, `runtime.alignment_evidence` owns capsule integrity, and
229
+ `application.readiness` orchestrates reuse/execution. `application.materialize`
230
+ materializes deterministic full-word timing from accepted immutable evidence without
231
+ importing or executing an aligner.
232
+
233
+ ## Optional Dependency and Registry Behavior
234
+
235
+ `BackendRegistry` stores immutable dotted factories. Construction imports only the
236
+ selected backend module and validates the result against `AlignmentBackend`. An absent
237
+ optional runtime raises an actionable `BackendUnavailableError` only when selected.
238
+ One backend's absence does not affect backend-neutral imports or sibling construction.
239
+
240
+ ## Architecture Verification
241
+
242
+ Release gates enforce:
243
+
244
+ - AST/import-boundary tests for prohibited directions and heavyweight imports;
245
+ - seven Import Linter contracts;
246
+ - backend sibling isolation;
247
+ - public-library tests with fake/injected backends;
248
+ - optional dependency absence;
249
+ - focused implementation-to-test mapping;
250
+ - per-file and aggregate coverage.
251
+
252
+ The current observed gate is documented in [`docs/TESTING.md`](docs/TESTING.md).
253
+
254
+ ## Evolution Rules
255
+
256
+ Adding a backend should require a config, native input/execution/parser adapter, and
257
+ focused tests. It should not require changes to metadata I/O, common normalization,
258
+ first/last extraction, benchmark metrics, or the CLI beyond registry composition.
259
+ Shared helpers are introduced only when semantics are genuinely identical.
README.md CHANGED
@@ -1,59 +1,132 @@
1
  ---
2
  license: apache-2.0
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3
  ---
4
 
5
  # Speech-Text Data Aligners
6
 
7
- A library-first framework that uses a known, authoritative English transcript to
8
- estimate word timing in WAV audio. It performs forced alignment, not ASR
9
- transcription, and never replaces `en_transcript` with recognized text.
 
10
 
11
- ## Features
 
 
12
 
13
- - One backend-neutral Python contract for NeMo, MFA, and WhisperX.
14
- - Reversible normalization and monotonic mapping back to original transcript words.
15
- - Validated, non-fabricated timestamps with structured failures and provenance.
 
 
 
 
 
 
 
16
  - Source-preserving, deterministic CSV annotation with atomic publication.
17
- - Reusable annotation and cross-backend benchmark services independent of the CLI.
18
- - Lazy, isolated optional backends; ordinary tests require no model, GPU, or network.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
19
 
20
- ## Supported Backends
 
 
 
 
21
 
22
- | Backend | Implementation | Empirical status |
23
- |---|---|---|
24
- | NVIDIA NeMo Forced Aligner (NFA) | Accepted | Real NVIDIA A10G smoke, five-row benchmark, and strict 36-row candidate passed |
25
- | Montreal Forced Aligner (MFA) 3.x | Accepted | **Selected final backend**; five-row and strict full-36 validation passed |
26
- | WhisperX 3.8.6 | Accepted | Five-row and strict full-36 validation passed; accepted alternative |
27
 
28
- Implementation acceptance and empirical validation are separate. See
29
- [Project Status](docs/PROJECT_STATUS.md) for the evidence boundary.
 
 
 
30
 
31
- ## Input and Output
 
 
 
32
 
33
- Input metadata must contain:
 
 
34
 
35
  ```text
36
  line_id,audio_id,en_transcript,ru_translation
37
  ```
38
 
39
- Audio is resolved as `<audio_dir>/<audio_id>.wav`. Annotation preserves all source
40
- columns, values, and row order, then appends exactly:
41
 
42
  ```text
43
- first_word_text,first_word_start_s,first_word_end_s,
44
- last_word_text,last_word_start_s,last_word_end_s
45
  ```
46
 
47
- `en_transcript` is authoritative. Backends may create internal reference
48
- representations, but independent ASR output cannot become the transcript.
49
 
50
- ## Quick Start
 
 
51
 
52
- Use the pinned environment definitions in [`environments/`](environments/) and keep
53
- backend runtimes isolated. Detailed setup and execution paths are in
54
- [Workflows](docs/WORKFLOWS.md).
55
 
56
- Minimal Python API:
 
 
57
 
58
  ```python
59
  from pathlib import Path
@@ -69,61 +142,146 @@ run = annotate_metadata(
69
  backend=MfaBackend(MfaConfig(environment_name="speech-aligners-mfa")),
70
  options=AnnotationOptions(strict=False),
71
  )
 
72
  print(run.receipt.success_count, run.receipt.failure_count)
73
  ```
74
 
75
- Minimal CLI:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
76
 
77
  ```bash
78
- PYTHONPATH=src python -m speech_text_data_aligners.cli.main annotate \
79
  metadata.csv wavs annotated.csv \
80
  --backend mfa \
81
- --backend-config-json '{"environment_name":"speech-aligners-mfa"}'
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
82
  ```
83
 
84
- Use `--strict` to abort publication on the first failure. Without it, failed source
85
- rows are preserved with six blank alignment cells and a diagnostics JSON artifact.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
86
 
87
  ## Documentation
88
 
89
- - [Project status](docs/PROJECT_STATUS.md)
90
- - [Architecture](docs/ARCHITECTURE.md)
91
- - [Workflows](docs/WORKFLOWS.md)
92
- - [Aligner engineering contract](docs/ALIGNER_ENGINEERING_CONTRACT.md)
93
- - [Modular architecture contract](docs/MODULAR_ARCHITECTURE_CONTRACT.md)
94
- - [Stage 5 final acceptance report](docs/STAGE_5_FINAL_ACCEPTANCE_REPORT.md)
95
- - [Post-final three-aligner comparison](docs/POSTFINAL_THREE_ALIGNER_COMPARISON.md)
96
- - [Final aligner selection](docs/FINAL_ALIGNER_SELECTION.md)
97
- - [W1-T offline semantic-readiness timing](docs/W1_T_SEMANTIC_READINESS.md)
98
 
99
- ## Current Validation Status
 
 
 
 
 
 
100
 
101
- Stages 0–5 are accepted, the NVIDIA-host NeMo closure is complete, and the post-final
102
- MFA extension has closed the full-corpus comparison. All three backends now have successful
103
- 36/36 candidates. On the five shared manual-gold rows, first-word-end MAE is 0.0162 s for
104
- MFA, 0.0154 s for WhisperX, and 0.2210 s for NeMo.
105
 
106
- MFA is the selected operational backend after the symmetric 36-row comparison. The earlier
107
- WhisperX selection was made before MFA had a strict full-36 result and is superseded by the
108
- 2026-09-06 post-final review. The canonical output is
109
- `results/final/final-annotated-36.csv` (36/36 rows; SHA-256 `37bcb85fc22fb142dae40453b90dd3b427bbfd657861bee7529cd433915535a8`).
 
110
 
111
- Only first-word text and first-word end currently have manual gold. First-word start,
112
- last-word start, and last-word end have structural validation and pairwise comparison,
113
- but no manual ground-truth validation. Pairwise agreement remains review evidence, not
114
- ground truth.
115
 
116
- ## License & Copyright
117
 
118
- Licensed under the [Apache License 2.0](LICENSE). The
119
- [official license text](https://www.apache.org/licenses/LICENSE-2.0) is also available
120
- from the Apache Software Foundation.
121
 
122
  Copyright (c) 2026 VertoX-AI. All rights reserved.
123
 
124
  ## Author
125
 
126
- Patrick Lumbantobing — [VertoX-AI](https://huggingface.co/vertox-ai)
127
 
128
  ## Citation
129
 
@@ -137,13 +295,13 @@ Patrick Lumbantobing — [VertoX-AI](https://huggingface.co/vertox-ai)
137
  }
138
  ```
139
 
140
- ## Acknowledgments
 
 
 
 
 
 
 
141
 
142
- This project builds on [NVIDIA NeMo and NeMo Forced Aligner](https://docs.nvidia.com/nemo/speech/nightly/tools/nemo_forced_aligner.html),
143
- [Montreal Forced Aligner](https://montreal-forced-aligner.readthedocs.io/en/latest/),
144
- [WhisperX](https://github.com/m-bain/whisperX),
145
- [PyTorch](https://pytorch.org/),
146
- [Torchaudio](https://docs.pytorch.org/audio/stable/index.html), and
147
- [Hugging Face](https://huggingface.co/). WhisperX uses Torchaudio's selected
148
- [WAV2VEC2_ASR_BASE_960H bundle](https://docs.pytorch.org/audio/2.8.0/generated/torchaudio.pipelines.WAV2VEC2_ASR_BASE_960H.html).
149
  Acknowledgment does not imply endorsement by any upstream project or organization.
 
1
  ---
2
  license: apache-2.0
3
+ language: en
4
+ tags:
5
+ - audio
6
+ - speech
7
+ - forced-alignment
8
+ - reference-text-alignment
9
+ - speech-processing
10
+ - mfa
11
+ - nemo
12
+ - whisperx
13
+ - python
14
+ - metric:mae
15
+ model-index:
16
+ - name: speech-text-data-aligners
17
+ results:
18
+ - task:
19
+ type: forced-alignment
20
+ name: Forced alignment
21
+ dataset:
22
+ type: local
23
+ name: five-row-first-word-manual-gold
24
+ metrics:
25
+ - type: mae
26
+ value: 0.0162
27
+ name: MFA first-word-end MAE in seconds
28
+ - type: mae
29
+ value: 0.0154
30
+ name: WhisperX first-word-end MAE in seconds
31
+ - type: mae
32
+ value: 0.2210
33
+ name: NeMo first-word-end MAE in seconds
34
  ---
35
 
36
  # Speech-Text Data Aligners
37
 
38
+ Speech-Text Data Aligners (STA) is a library-first framework for forced alignment of
39
+ WAV audio against a known, authoritative English transcript. It estimates when the
40
+ supplied words occur in audio. It is not ASR and never replaces `en_transcript` with a
41
+ recognized transcript.
42
 
43
+ The repository provides one backend-neutral Python contract for Montreal Forced
44
+ Aligner (MFA), NVIDIA NeMo Forced Aligner (NFA), and WhisperX, plus deterministic
45
+ annotation, benchmarking, and offline semantic-readiness services.
46
 
47
+ This repository is a software framework and comparative evaluation package, not a
48
+ single neural checkpoint. [`config.json`](config.json) is its machine-readable
49
+ project/model manifest; it is not a Transformers `AutoConfig` file.
50
+
51
+ ## Highlights
52
+
53
+ - Authoritative reference-text alignment with no silent ASR substitution.
54
+ - Reversible lexical mapping back to exact source transcript words.
55
+ - Strict finite, non-negative, monotonic, non-overlapping, WAV-bounded timestamps.
56
+ - No clamping, interpolation, extrapolation, or fabricated success/timing.
57
  - Source-preserving, deterministic CSV annotation with atomic publication.
58
+ - Python-callable services; the CLI delegates to those services.
59
+ - Typed backend configuration, provenance, diagnostics, and optional native artifacts.
60
+ - Lazy backend isolation: missing NeMo/MFA/WhisperX dependencies do not break imports.
61
+ - Deterministic unit, architecture, header, manifest, and per-file coverage gates.
62
+
63
+ ## Model and Configuration Matrix
64
+
65
+ | Backend | Pinned alignment model | Runtime baseline | Default device | Current role |
66
+ |---|---|---|---|---|
67
+ | MFA | `MontrealCorpusTools/english_mfa@1009e3c6c689f10875146a5e4e5c2233f8eaf922` (`english_us_mfa`, model 3.3.0) | MFA 3.4.2 | CPU | **Selected operational backend** |
68
+ | WhisperX | `WAV2VEC2_ASR_BASE_960H` / Torchaudio 2.8.0 bundle | WhisperX 3.8.6 | CPU | Accepted alternative |
69
+ | NeMo / NFA | `stt_en_fastconformer_hybrid_large_pc` | NeMo 3.0.0 / source `fd6a877539710e2b98f28c43272ff81312f83417` | CUDA | Accepted; not selected operationally |
70
+
71
+ All supported configuration fields, defaults, invariants, and JSON examples are in
72
+ [Configuration](docs/CONFIGURATION.md). Compact model, schema, and evaluation metadata
73
+ are in [`config.json`](config.json).
74
+
75
+ ## Installation
76
+
77
+ Use Python 3.12 or newer. Keep each heavyweight backend in its dedicated environment.
78
+ The repository contains a deterministic development environment and separate backend
79
+ environment specifications:
80
 
81
+ ```bash
82
+ conda env create -f environments/core-dev.yml
83
+ conda activate speech-aligners-core-dev
84
+ python -m pip install -e .
85
+ ```
86
 
87
+ An existing environment named `speech-text-data-aligners` is also suitable when it
88
+ contains the packages pinned in [`requirements.txt`](requirements.txt):
 
 
 
89
 
90
+ ```bash
91
+ conda deactivate
92
+ conda activate speech-text-data-aligners
93
+ python -m pip install -e . --no-deps
94
+ ```
95
 
96
+ Backend environments are created separately from
97
+ [`environments/mfa.yml`](environments/mfa.yml), [`environments/nemo.yml`](environments/nemo.yml),
98
+ and [`environments/whisperx.yml`](environments/whisperx.yml). The parent CLI process
99
+ invokes the selected isolated runtime; see [Configuration](docs/CONFIGURATION.md).
100
 
101
+ ## Data Contract
102
+
103
+ Input metadata must contain these columns:
104
 
105
  ```text
106
  line_id,audio_id,en_transcript,ru_translation
107
  ```
108
 
109
+ Audio must be available at `<audio_dir>/<audio_id>.wav`. Annotation preserves every
110
+ source header, cell value, CSV row, and row position, then appends exactly:
111
 
112
  ```text
113
+ first_word_text,first_word_start_s,first_word_end_s,last_word_text,last_word_start_s,last_word_end_s
 
114
  ```
115
 
116
+ For a one-word utterance, first and last fields come from the same aligned word.
117
+ Successful timing must satisfy:
118
 
119
+ ```text
120
+ 0 <= first_start <= first_end <= last_start <= last_end <= audio_duration
121
+ ```
122
 
123
+ Equality is allowed at backend resolution limits. NaN, infinity, negative values,
124
+ reversed/overlapping intervals, incomplete lexical coverage, and out-of-audio values
125
+ fail validation.
126
 
127
+ ## Python Usage
128
+
129
+ The root package exposes the stable backend-neutral API:
130
 
131
  ```python
132
  from pathlib import Path
 
142
  backend=MfaBackend(MfaConfig(environment_name="speech-aligners-mfa")),
143
  options=AnnotationOptions(strict=False),
144
  )
145
+
146
  print(run.receipt.success_count, run.receipt.failure_count)
147
  ```
148
 
149
+ Custom or test backends can implement `AlignmentBackend`; application code receives
150
+ them by dependency injection and does not branch on backend names.
151
+
152
+ For one backend:
153
+
154
+ ```python
155
+ from pathlib import Path
156
+
157
+ from speech_text_data_aligners import benchmark_backend
158
+ from speech_text_data_aligners.backends.mfa.backend import MfaBackend
159
+ from speech_text_data_aligners.backends.mfa.config import MfaConfig
160
+
161
+ report = benchmark_backend(
162
+ backend=MfaBackend(MfaConfig()),
163
+ metadata_csv=Path("metadata.csv"),
164
+ reference_csv=Path("first-word-gold.csv"),
165
+ audio_dir=Path("wavs"),
166
+ )
167
+ ```
168
+
169
+ Use `benchmark_backends` for an insertion-ordered mapping of multiple backend
170
+ instances. Specialized offline readiness services are documented in
171
+ [W1-T semantic readiness](docs/W1_T_SEMANTIC_READINESS.md).
172
+
173
+ ## CLI Usage
174
+
175
+ After installation, `speech-text-aligners` and
176
+ `python -m speech_text_data_aligners.cli.main` are equivalent entry points.
177
+
178
+ Annotate with the selected MFA environment:
179
 
180
  ```bash
181
+ speech-text-aligners annotate \
182
  metadata.csv wavs annotated.csv \
183
  --backend mfa \
184
+ --backend-config-json '{"environment_name":"speech-aligners-mfa"}' \
185
+ --receipt annotated.receipt.json
186
+ ```
187
+
188
+ Benchmark multiple backends. Benchmark configuration is one JSON object keyed by the
189
+ values passed to `--backend`:
190
+
191
+ ```bash
192
+ speech-text-aligners benchmark \
193
+ metadata.csv first-word-gold.csv wavs \
194
+ --backend mfa \
195
+ --backend whisperx \
196
+ --backend-config-json '{"mfa":{"environment_name":"speech-aligners-mfa"},"whisperx":{"python_executable":"/path/to/whisperx/bin/python","device":"cpu"}}' \
197
+ --output-base reports/comparison
198
+ ```
199
+
200
+ Run offline semantic readiness with an explicit provider/evidence catalog:
201
+
202
+ ```bash
203
+ speech-text-aligners readiness \
204
+ source_metadata_en_only.csv boundaries.jsonl wavs results/w1t/new-attempt \
205
+ --catalog /absolute/path/to/catalog.json
206
  ```
207
 
208
+ Run `speech-text-aligners <command> --help` for the authoritative argument list.
209
+ Detailed workflows and output artifacts are in [Workflows](docs/WORKFLOWS.md).
210
+
211
+ ## Failure and Publication Policy
212
+
213
+ - `--strict` aborts on the first row/backend failure and publishes no annotated CSV.
214
+ - Without `--strict`, failed rows retain their source cells, leave all six alignment
215
+ cells blank, and are recorded in a deterministic diagnostics JSON file.
216
+ - Existing output artifacts require `--overwrite`.
217
+ - Annotation writes a complete temporary CSV, flushes and fsyncs it, then atomically
218
+ publishes it. Input/output aliases are rejected before backend execution.
219
+ - Benchmarks retain structured failure observations and score only available
220
+ references; failures never become fabricated timestamps.
221
+
222
+ ## Evaluation Summary
223
+
224
+ Manual gold exists for only `first_word_text` and `first_word_end_s` on five rows.
225
+
226
+ | Backend | Success | Exact first-word text | First-word-end MAE | Median | Maximum |
227
+ |---|---:|---:|---:|---:|---:|
228
+ | MFA | 5/5 | 5/5 | **0.0162 s** | **0.0110 s** | **0.0300 s** |
229
+ | WhisperX | 5/5 | 5/5 | **0.0154 s** | **0.0140 s** | **0.0310 s** |
230
+ | NeMo | 5/5 | 5/5 | **0.2210 s** | **0.0100 s** | **0.9200 s** |
231
+
232
+ MFA and WhisperX are effectively tied on this sparse gold; the 0.0008 s MAE difference
233
+ is not decision-grade. All three backends produced 36/36 structurally valid candidates.
234
+ At a 0.250 s material-disagreement threshold, MFA/WhisperX differs on 7 rows, MFA/NeMo
235
+ on 28, and WhisperX/NeMo on 32.
236
+
237
+ Only first-word text and first-word end have manual ground truth. First-word start and
238
+ both last-word boundaries have structural validation and pairwise comparison, but no
239
+ manual accuracy evidence. Pairwise agreement is diagnostic, not ground truth. Runtime
240
+ observations used different hosts, devices, and cache states and are secondary.
241
+
242
+ See [Evaluation](docs/EVALUATION.md) for metric definitions, model/configuration
243
+ coverage, full pairwise tables, W1-T evidence, caveats, and artifact paths.
244
+
245
+ ## Engineering Status
246
+
247
+ The current checkout passes 635 deterministic tests with 3 heavyweight integration
248
+ tests skipped, 98.74% aggregate line coverage, all individual implementation-file
249
+ coverage gates, Ruff, strict mypy, and all 7 import-linter contracts. The current
250
+ lowest implementation file is 93.33%. Commands and interpretation are in
251
+ [Testing and Quality](docs/TESTING.md).
252
 
253
  ## Documentation
254
 
255
+ Current guidance:
 
 
 
 
 
 
 
 
256
 
257
+ - [Architecture](ARCHITECTURE.md)
258
+ - [Configuration](docs/CONFIGURATION.md)
259
+ - [Evaluation](docs/EVALUATION.md)
260
+ - [Workflows](docs/WORKFLOWS.md)
261
+ - [Testing and Quality](docs/TESTING.md)
262
+ - [Project Status](docs/PROJECT_STATUS.md)
263
+ - [Documentation Map](docs/DOCUMENTATION.md)
264
 
265
+ Contracts and evidence:
 
 
 
266
 
267
+ - [Aligner Engineering Contract](docs/ALIGNER_ENGINEERING_CONTRACT.md)
268
+ - [Modular Architecture Contract](docs/MODULAR_ARCHITECTURE_CONTRACT.md)
269
+ - [Final Aligner Selection](docs/FINAL_ALIGNER_SELECTION.md)
270
+ - [Post-final Three-Aligner Comparison](docs/POSTFINAL_THREE_ALIGNER_COMPARISON.md)
271
+ - [W1-T Semantic Readiness](docs/W1_T_SEMANTIC_READINESS.md)
272
 
273
+ Historical stage reports remain unchanged as implementation and acceptance provenance;
274
+ they are not the primary operating guide.
 
 
275
 
276
+ ## License
277
 
278
+ Licensed under the [Apache License 2.0](LICENSE).
 
 
279
 
280
  Copyright (c) 2026 VertoX-AI. All rights reserved.
281
 
282
  ## Author
283
 
284
+ Patrick Lumbantobing - [VertoX-AI](https://huggingface.co/vertox-ai)
285
 
286
  ## Citation
287
 
 
295
  }
296
  ```
297
 
298
+ ## References
299
+
300
+ - [Hugging Face model cards](https://huggingface.co/docs/hub/en/model-cards)
301
+ - [NVIDIA NeMo Forced Aligner](https://docs.nvidia.com/nemo/speech/nightly/tools/nemo_forced_aligner.html)
302
+ - [Montreal Forced Aligner](https://montreal-forced-aligner.readthedocs.io/en/latest/)
303
+ - [WhisperX](https://github.com/m-bain/whisperX)
304
+ - [PyTorch](https://pytorch.org/)
305
+ - [Torchaudio WAV2VEC2_ASR_BASE_960H](https://docs.pytorch.org/audio/2.8.0/generated/torchaudio.pipelines.WAV2VEC2_ASR_BASE_960H.html)
306
 
 
 
 
 
 
 
 
307
  Acknowledgment does not imply endorsement by any upstream project or organization.
SKILLS.md CHANGED
@@ -83,3 +83,19 @@ Run all executable backends on the exact same input subset. Compare at least:
83
  - runtime per utterance and total runtime where practical.
84
 
85
  Do not infer accuracy of unannotated fields from backend agreement alone.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
83
  - runtime per utterance and total runtime where practical.
84
 
85
  Do not infer accuracy of unannotated fields from backend agreement alone.
86
+
87
+ ## Skill: documentation and manifest integrity
88
+
89
+ Treat documentation as a tested interface. Keep the Hugging Face-style README,
90
+ root `ARCHITECTURE.md`, focused usage/config/evaluation/testing guides, and
91
+ `config.json` aligned with actual source behavior and retained evidence.
92
+
93
+ Use `config.json` as a versioned project/model manifest, not as a Transformers
94
+ checkpoint configuration or a direct CLI backend-config payload. Add deterministic
95
+ tests that bind its package version, schema fields, registered backends, model
96
+ identities, and canonical result hashes to source/evidence values.
97
+
98
+ Separate current operating guidance from contracts, empirical evidence, and historical
99
+ acceptance reports. Link all local documents from a maintained documentation map,
100
+ validate local Markdown links, and run every documented syntax/help/quality command.
101
+ Never turn pairwise agreement or structural success into an accuracy claim.
config.json ADDED
@@ -0,0 +1,162 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "schema_version": "speech-text-data-aligners/config.v1",
3
+ "project": {
4
+ "name": "speech-text-data-aligners",
5
+ "version": "0.1.0",
6
+ "package": "speech_text_data_aligners",
7
+ "python": ">=3.12",
8
+ "entrypoint": "speech-text-aligners",
9
+ "license": "Apache-2.0",
10
+ "task": "forced-alignment",
11
+ "languages": [
12
+ "en"
13
+ ],
14
+ "author": "Patrick Lumbantobing, VertoX-AI",
15
+ "repository": "https://huggingface.co/vertox-ai/speech-text-data-aligners"
16
+ },
17
+ "data_contract": {
18
+ "transcript_column": "en_transcript",
19
+ "source_columns": [
20
+ "line_id",
21
+ "audio_id",
22
+ "en_transcript",
23
+ "ru_translation"
24
+ ],
25
+ "audio_path_template": "<audio_dir>/<audio_id>.wav",
26
+ "alignment_columns": [
27
+ "first_word_text",
28
+ "first_word_start_s",
29
+ "first_word_end_s",
30
+ "last_word_text",
31
+ "last_word_start_s",
32
+ "last_word_end_s"
33
+ ],
34
+ "timestamp_unit": "seconds",
35
+ "authoritative_transcript": true
36
+ },
37
+ "backend_registry": {
38
+ "operational_selection": "mfa",
39
+ "models": {
40
+ "mfa": {
41
+ "selected": true,
42
+ "config_class": "speech_text_data_aligners.backends.mfa.config.MfaConfig",
43
+ "runtime": {
44
+ "package": "montreal-forced-aligner",
45
+ "version": "3.4.2",
46
+ "source_revision": "d2dc283bd79667e086b1f93050a1855349f63f1f",
47
+ "kaldi_version": "5.5.1172",
48
+ "kalpy_version": "0.10.5",
49
+ "openfst_version": "1.8.4"
50
+ },
51
+ "model": {
52
+ "id": "MontrealCorpusTools/english_mfa",
53
+ "revision": "1009e3c6c689f10875146a5e4e5c2233f8eaf922",
54
+ "version": "3.3.0",
55
+ "dialect": "english_us_mfa"
56
+ },
57
+ "observed_evidence": {
58
+ "model_tree_sha256": "486bf1be991b0590f3554040a16502d033237f63d5e426ca231b92b0a9829ada"
59
+ }
60
+ },
61
+ "whisperx": {
62
+ "selected": false,
63
+ "config_class": "speech_text_data_aligners.backends.whisperx.config.WhisperXConfig",
64
+ "runtime": {
65
+ "package": "whisperx",
66
+ "version": "3.8.6",
67
+ "source_revision": "3ccc17b8de34f305300f8a3fd3c9f76ba820c0d0",
68
+ "torch_version": "2.8.0",
69
+ "torchaudio_version": "2.8.0"
70
+ },
71
+ "model": {
72
+ "id": "WAV2VEC2_ASR_BASE_960H",
73
+ "revision": "torchaudio-2.8.0",
74
+ "language": "en"
75
+ },
76
+ "observed_evidence": {
77
+ "model_asset_sha256": "488fd4f16de84438ffc945334278c1b9fb9b7159a806c1080b16111a958c945d"
78
+ }
79
+ },
80
+ "nemo": {
81
+ "selected": false,
82
+ "config_class": "speech_text_data_aligners.backends.nemo.config.NemoConfig",
83
+ "runtime": {
84
+ "package": "nemo-toolkit",
85
+ "version": "3.0.0",
86
+ "source_revision": "fd6a877539710e2b98f28c43272ff81312f83417",
87
+ "align_script_sha256": "1cfaf9526b47037204b7874c4432e4d5d07364a7fca1b840052c370d46ee2f03"
88
+ },
89
+ "model": {
90
+ "id": "stt_en_fastconformer_hybrid_large_pc",
91
+ "revision": null,
92
+ "resolution": "pretrained_name_or_local_checkpoint",
93
+ "local_checkpoint_preferred": true
94
+ },
95
+ "observed_evidence": {
96
+ "model_sha256": "37afe6627816825af7646c02f9ada5dbf58fe870e7a9cc60fffbd8dd412c8b61"
97
+ }
98
+ }
99
+ }
100
+ },
101
+ "evaluation": {
102
+ "manual_gold": {
103
+ "rows": 5,
104
+ "fields": [
105
+ "first_word_text",
106
+ "first_word_end_s"
107
+ ],
108
+ "unvalidated_fields": [
109
+ "first_word_start_s",
110
+ "last_word_start_s",
111
+ "last_word_end_s"
112
+ ]
113
+ },
114
+ "results": [
115
+ {
116
+ "backend": "mfa",
117
+ "success": "5/5",
118
+ "exact_first_word_text": "5/5",
119
+ "first_word_end_mae_s": 0.0162,
120
+ "first_word_end_median_s": 0.0110,
121
+ "first_word_end_max_s": 0.0300,
122
+ "observed_runtime_s": 274.9569
123
+ },
124
+ {
125
+ "backend": "whisperx",
126
+ "success": "5/5",
127
+ "exact_first_word_text": "5/5",
128
+ "first_word_end_mae_s": 0.0154,
129
+ "first_word_end_median_s": 0.0140,
130
+ "first_word_end_max_s": 0.0310,
131
+ "observed_runtime_s": 28.5392
132
+ },
133
+ {
134
+ "backend": "nemo",
135
+ "success": "5/5",
136
+ "exact_first_word_text": "5/5",
137
+ "first_word_end_mae_s": 0.2210,
138
+ "first_word_end_median_s": 0.0100,
139
+ "first_word_end_max_s": 0.9200,
140
+ "observed_runtime_s": 13.7050
141
+ }
142
+ ],
143
+ "full_36_pairwise_material_rows_at_0_250_s": {
144
+ "mfa_vs_whisperx": 7,
145
+ "mfa_vs_nemo": 28,
146
+ "whisperx_vs_nemo": 32
147
+ },
148
+ "canonical_output": {
149
+ "path": "results/final/final-annotated-36.csv",
150
+ "sha256": "37bcb85fc22fb142dae40453b90dd3b427bbfd657861bee7529cd433915535a8"
151
+ }
152
+ },
153
+ "documentation": {
154
+ "readme": "README.md",
155
+ "architecture": "ARCHITECTURE.md",
156
+ "configuration": "docs/CONFIGURATION.md",
157
+ "evaluation": "docs/EVALUATION.md",
158
+ "workflows": "docs/WORKFLOWS.md",
159
+ "testing": "docs/TESTING.md",
160
+ "status": "docs/PROJECT_STATUS.md"
161
+ }
162
+ }
docs/ARCHITECTURE.md CHANGED
@@ -1,187 +1,5 @@
1
  # Architecture
2
 
3
- Speech-Text Data Aligners is a library-first forced-alignment framework. The core
4
- contract accepts WAV audio plus a known, authoritative transcript and returns
5
- backend-neutral timed words. The CLI is only a composition and presentation layer.
6
-
7
- The mandatory design rules are defined in the
8
- [modular architecture contract](MODULAR_ARCHITECTURE_CONTRACT.md) and the
9
- [aligner engineering contract](ALIGNER_ENGINEERING_CONTRACT.md).
10
-
11
- ## Layers and Dependency Direction
12
-
13
- ```text
14
- CLI --------------------> Application --------------------> Core
15
- | ^
16
- +----------> Data -------------+
17
- +----------> Benchmarking -----+
18
- +----------> Runtime ----------+
19
-
20
- CLI -- lazy composition --> Concrete Backends
21
- | |
22
- +------------> Core
23
- +------------> Runtime
24
- ```
25
-
26
- In compact form:
27
-
28
- ```text
29
- CLI -> Application -> Core
30
- -> Data
31
- -> Benchmarking
32
- -> Runtime
33
-
34
- Concrete Backends -> Core / Runtime
35
- ```
36
-
37
- The enforced consequences are:
38
-
39
- - Core does not import application, data, benchmarking, CLI, or concrete backends.
40
- - Data and benchmarking do not depend on concrete backends.
41
- - Application services receive an `AlignmentBackend` protocol instance; they do not
42
- branch on NeMo, MFA, or WhisperX names.
43
- - Concrete backend packages do not import one another.
44
- - The registry stores lazy dotted factories so unavailable optional runtimes do not
45
- break backend-neutral imports or sibling backends.
46
- - The CLI delegates annotation and benchmarking to reusable Python services.
47
-
48
- ## Current Package Tree
49
-
50
- ```text
51
- src/speech_text_data_aligners/
52
- |-- __init__.py public backend-neutral facade
53
- |-- application/
54
- | |-- annotate.py annotation orchestration and publication policy
55
- | `-- benchmark.py identical-request benchmark orchestration
56
- |-- backends/
57
- | |-- registry.py immutable lazy backend composition
58
- | |-- nemo/
59
- | | |-- backend.py neutral adapter
60
- | | |-- config.py typed configuration
61
- | | |-- manifest.py reference JSONL and output-manifest parsing
62
- | | |-- runner.py isolated NFA execution
63
- | | `-- ctm.py word CTM parsing and mapping
64
- | |-- mfa/
65
- | | |-- backend.py neutral adapter
66
- | | |-- config.py typed configuration
67
- | | |-- corpus.py temporary WAV/LAB corpus
68
- | | |-- runner.py isolated MFA execution
69
- | | `-- textgrid.py word-tier parsing and mapping
70
- | `-- whisperx/
71
- | |-- backend.py neutral adapter
72
- | |-- config.py typed configuration
73
- | |-- segments.py full-WAV authoritative segments
74
- | |-- provider.py isolated-worker transport
75
- | |-- worker.py lazy heavyweight native runtime
76
- | `-- parser.py native word-dictionary parsing and mapping
77
- |-- benchmarking/
78
- | |-- metrics.py sparse-gold and pairwise metrics
79
- | `-- reports.py JSON/CSV/Markdown report rendering
80
- |-- cli/
81
- | |-- main.py command dispatch
82
- | |-- annotate.py annotation argument adapter
83
- | `-- benchmark.py benchmark argument adapter
84
- |-- core/
85
- | |-- models.py immutable neutral values and provenance
86
- | |-- protocols.py injected backend contracts
87
- | |-- errors.py project exception taxonomy
88
- | |-- normalization.py lexical plans and monotonic reconciliation
89
- | |-- validation.py result and publication-path invariants
90
- | `-- extraction.py six first/last output fields
91
- |-- data/
92
- | |-- audio.py WAV resolution and lightweight probing
93
- | `-- metadata.py preserving, atomic CSV I/O
94
- `-- runtime/
95
- |-- process.py shell-free injected subprocess boundary
96
- |-- artifacts.py temporary and retained artifact lifecycle
97
- `-- receipts.py hashes and run provenance
98
- ```
99
-
100
- Package-marker `__init__.py` files not shown above are deliberately empty. The root
101
- `__init__.py` is the tested public facade.
102
-
103
- ## Backend Isolation and Native Containment
104
-
105
- ```text
106
- PARENT PROCESS
107
-
108
- Python caller or CLI
109
- |
110
- v
111
- backend-neutral application API
112
- |
113
- v
114
- AlignmentBackend protocol
115
- |
116
- v
117
- concrete adapter (NeMo | MFA | WhisperX)
118
- |
119
- | injected process/provider boundary
120
- v
121
- +------------------------------------------------------+
122
- | ISOLATED NATIVE RUNTIME |
123
- | NeMo NFA process | MFA process | WhisperX worker |
124
- | Heavy optional dependencies and model lifecycle |
125
- +------------------------------------------------------+
126
- |
127
- | native files / native dictionaries
128
- v
129
- adapter-local builder, parser, and reconciliation
130
- |
131
- | only neutral records cross this point
132
- v
133
- AlignmentResult | AlignmentFailure | diagnostics | provenance
134
- ```
135
-
136
- NeMo manifests, output manifests, and CTM rows remain inside
137
- `backends.nemo`. MFA corpus labels and TextGrid objects remain inside
138
- `backends.mfa`. WhisperX segment payloads and native result dictionaries remain
139
- inside `backends.whisperx`. NeMo CTM, MFA TextGrid, and WhisperX native dictionaries
140
- never escape their adapters; callers receive only backend-neutral models.
141
-
142
- The parent process does not import heavyweight NeMo, MFA, WhisperX, Torch, or
143
- Torchaudio packages. NeMo and MFA run through the shared shell-free process boundary.
144
- WhisperX runs in a dedicated worker that imports its heavy stack only after invocation.
145
-
146
- ## Transcript-to-Result Data Flow
147
-
148
- ```text
149
- authoritative transcript
150
- |
151
- v
152
- backend-specific reference representation
153
- |
154
- v
155
- native timed words
156
- |
157
- v
158
- shared monotonic reconciliation
159
- |
160
- v
161
- shared validation
162
- |
163
- v
164
- AlignmentResult
165
- ```
166
-
167
- The authoritative `en_transcript` remains unchanged throughout. A backend may
168
- normalize or expand text only through a reversible plan that retains ownership of
169
- original lexical words. Shared reconciliation requires complete monotonic coverage,
170
- and shared validation rejects non-finite, negative, reversed, overlapping, incomplete,
171
- or out-of-audio timestamps. It never clamps, interpolates, or invents a timestamp.
172
-
173
- ## Reuse and Composition
174
-
175
- The root facade exposes reusable annotation and benchmark services. Applications may
176
- inject any object satisfying `AlignmentBackend`, including a lightweight test or
177
- custom backend. The default registry is a convenience for the CLI, not a dependency of
178
- the application layer.
179
-
180
- Benchmarking consumes the same neutral results as annotation. Native parsing,
181
- transcript mapping, CSV I/O, metric calculation, report rendering, and process
182
- execution each have one owning layer, so adding another adapter does not require
183
- changes to established backend-neutral behavior.
184
-
185
- Architecture is enforced by deterministic AST tests and Import Linter contracts. The
186
- accepted audit and complete dependency evidence are recorded in the
187
- [Stage 5 final acceptance report](STAGE_5_FINAL_ACCEPTANCE_REPORT.md).
 
1
  # Architecture
2
 
3
+ The current architecture document is maintained at
4
+ [`../ARCHITECTURE.md`](../ARCHITECTURE.md). This path remains available for older
5
+ links and historical reports.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
docs/CONFIGURATION.md ADDED
@@ -0,0 +1,282 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Configuration
2
+
3
+ This page is the authoritative configuration reference for Speech-Text Data Aligners.
4
+ Typed backend dataclasses validate configuration before model or process execution.
5
+ Unknown keys and invalid primitive types are rejected rather than ignored.
6
+
7
+ ## Manifest Versus Runtime Configuration
8
+
9
+ [`config.json`](../config.json) is a versioned machine-readable project/model manifest.
10
+ It records package identity, the data contract, registered backends, pinned model
11
+ identity, evaluation scope, and documentation paths. It is validated by
12
+ `tests/quality/test_config_manifest.py`.
13
+
14
+ The manifest is not passed directly to `--backend-config-json`. Runtime configuration
15
+ has these CLI shapes:
16
+
17
+ | Command | `--backend-config-json` shape |
18
+ |---|---|
19
+ | `annotate` | One selected backend's configuration object, for example `{"environment_name":"speech-aligners-mfa"}` |
20
+ | `benchmark` | One outer object keyed by each selected backend, for example `{"mfa":{...},"whisperx":{...}}` |
21
+ | `readiness` | Provider configuration lives under each catalog `backends.<name>.config` value |
22
+
23
+ All path-valued JSON strings are converted to `pathlib.Path`. `null` is accepted only
24
+ for fields documented as optional or nullable.
25
+
26
+ ## Environment Isolation
27
+
28
+ | Runtime | Specification | Default selector | Notes |
29
+ |---|---|---|---|
30
+ | Core development | `environments/core-dev.yml` | `speech-aligners-core-dev` | Deterministic tests, lint, type, coverage, and import checks |
31
+ | MFA | `environments/mfa.yml` | `speech-aligners-mfa` | Dedicated CPU Conda runtime |
32
+ | NeMo | `environments/nemo.yml` | No default environment name | Requires explicit pinned source/script and local or registry model |
33
+ | WhisperX | `environments/whisperx.yml` | Current Python by default | Dedicated CPU runtime is recommended |
34
+
35
+ MFA and NeMo execute through an isolated process boundary. WhisperX executes in a
36
+ dedicated worker that imports Torch/Torchaudio/WhisperX only after invocation. Heavy
37
+ backend packages are never imported by core, data, benchmarking, or application layers.
38
+
39
+ ## Common Artifact Policy
40
+
41
+ MFA, NeMo, and WhisperX share these artifact semantics:
42
+
43
+ | `artifact_policy` | Behavior |
44
+ |---|---|
45
+ | `never` | Remove adapter workspaces; default |
46
+ | `on_failure` | Retain native workspace only after failure |
47
+ | `always` | Retain native workspace on success or failure |
48
+
49
+ When the policy is not `never`, `retention_root` is required. `run_id` must be one safe
50
+ path component. `temporary_root` is an optional parent for unique workspaces.
51
+
52
+ Retained native artifacts are diagnostic evidence. They do not replace the
53
+ backend-neutral result, and native CTM/TextGrid/WhisperX structures do not cross the
54
+ adapter interface.
55
+
56
+ ## MFA Configuration
57
+
58
+ Python type: `speech_text_data_aligners.backends.mfa.config.MfaConfig`.
59
+
60
+ | Field | Type | Default | Meaning |
61
+ |---|---|---|---|
62
+ | `conda_executable` | string | `conda` | Conda command or executable path |
63
+ | `environment_name` | string/null | `speech-aligners-mfa` | Dedicated environment name |
64
+ | `environment_prefix` | path/null | `null` | Dedicated environment prefix |
65
+ | `mfa_executable` | string | `mfa` | Executable inside the selected environment |
66
+ | `model_id` | string | `MontrealCorpusTools/english_mfa` | Hugging Face MFA model repository, without revision |
67
+ | `model_revision` | string | `1009e3c6c689f10875146a5e4e5c2233f8eaf922` | Required 40-character lowercase commit |
68
+ | `model_version` | string | `3.3.0` | Model bundle version recorded in provenance |
69
+ | `dialect` | string | `english_us_mfa` | Dictionary/G2P dialect |
70
+ | `use_g2p` | boolean | `true` | Generate pronunciations for dictionary OOVs |
71
+ | `expected_mfa_version` | string | `3.4.2` | Required MFA distribution |
72
+ | `expected_kaldi_version` | string | `5.5.1172` | Required Kaldi distribution |
73
+ | `expected_kalpy_version` | string | `0.10.5` | Required Kalpy distribution |
74
+ | `expected_openfst_version` | string | `1.8.4` | Required OpenFst distribution |
75
+ | `expected_model_sha256` | string/null | `null` | Optional lowercase model-tree SHA-256 |
76
+ | `num_jobs` | positive integer | `1` | MFA worker process count |
77
+ | `timeout_s` | positive finite number | `3600.0` | Timeout for probe and alignment commands |
78
+ | `artifact_policy` | enum | `never` | Native artifact retention policy |
79
+ | `temporary_root` | path/null | `null` | Parent for temporary workspaces |
80
+ | `retention_root` | path/null | `null` | Required parent when retention is enabled |
81
+ | `run_id` | string | `mfa-run` | Safe retained-workspace component |
82
+
83
+ Exactly one of `environment_name` and `environment_prefix` must be set. When selecting
84
+ a prefix from CLI JSON, use `"environment_name": null` explicitly. The adapter always
85
+ uses long TextGrid output and a commit-qualified model; those safety choices are not
86
+ configurable.
87
+
88
+ Minimal CLI JSON:
89
+
90
+ ```json
91
+ {
92
+ "environment_name": "speech-aligners-mfa",
93
+ "num_jobs": 1,
94
+ "artifact_policy": "never"
95
+ }
96
+ ```
97
+
98
+ Failure-only artifact example:
99
+
100
+ ```json
101
+ {
102
+ "environment_prefix": "/opt/conda/envs/speech-aligners-mfa",
103
+ "environment_name": null,
104
+ "artifact_policy": "on_failure",
105
+ "retention_root": "artifacts/mfa",
106
+ "run_id": "metadata-run-001"
107
+ }
108
+ ```
109
+
110
+ ## NeMo / NFA Configuration
111
+
112
+ Python type: `speech_text_data_aligners.backends.nemo.config.NemoConfig`.
113
+
114
+ | Field | Type | Default | Meaning |
115
+ |---|---|---|---|
116
+ | `align_script` | path | Required | Exact `tools/nemo_forced_aligner/align.py` |
117
+ | `python_executable` | path | Current interpreter | Python in the isolated NeMo runtime |
118
+ | `pretrained_name` | string/null | `stt_en_fastconformer_hybrid_large_pc` | Registry model name |
119
+ | `model_path` | path/null | `null` | Local `.nemo` checkpoint |
120
+ | `model_id` | string | `stt_en_fastconformer_hybrid_large_pc` | Stable model identity for provenance |
121
+ | `model_revision` | string/null | `null` | Registry/source revision when known |
122
+ | `expected_model_sha256` | string/null | `null` | Required local checkpoint SHA-256 |
123
+ | `expected_nemo_version` | string | `3.0.0` | Required NeMo distribution |
124
+ | `nemo_source_revision` | string | `fd6a877539710e2b98f28c43272ff81312f83417` | Pinned Speech checkout |
125
+ | `expected_align_script_sha256` | string | `1cfaf9526b47037204b7874c4432e4d5d07364a7fca1b840052c370d46ee2f03` | Required entry-point hash |
126
+ | `transcribe_device` | string | `cuda` | Torch device for acoustic inference |
127
+ | `viterbi_device` | string | `cuda` | Torch device for Viterbi alignment |
128
+ | `batch_size` | positive integer | `1` | NFA batch size |
129
+ | `use_local_attention` | boolean | `true` | Request supported local attention |
130
+ | `timeout_s` | positive finite number | `3600.0` | Timeout for probe and alignment |
131
+ | `save_ass` | boolean | `false` | Retain optional ASS output in addition to CTM |
132
+ | `artifact_policy` | enum | `never` | Native artifact retention policy |
133
+ | `temporary_root` | path/null | `null` | Parent for temporary workspaces |
134
+ | `retention_root` | path/null | `null` | Required parent when retention is enabled |
135
+ | `run_id` | string | `nemo-run` | Safe retained-workspace component |
136
+
137
+ Exactly one of `pretrained_name` and `model_path` must be set. When using a local
138
+ checkpoint from CLI JSON, set `"pretrained_name": null` explicitly.
139
+ `expected_model_sha256` is valid only with `model_path`.
140
+
141
+ Local-checkpoint example:
142
+
143
+ ```json
144
+ {
145
+ "align_script": "/opt/runtimes/nemo-speech-v3.0.0/tools/nemo_forced_aligner/align.py",
146
+ "python_executable": "/opt/runtimes/nemo-speech-v3.0.0/.venv/bin/python",
147
+ "pretrained_name": null,
148
+ "model_path": "/opt/models/stt_en_fastconformer_hybrid_large_pc.nemo",
149
+ "expected_model_sha256": "<lowercase-64-character-sha256>",
150
+ "transcribe_device": "cuda",
151
+ "viterbi_device": "cuda",
152
+ "artifact_policy": "on_failure",
153
+ "retention_root": "artifacts/nemo",
154
+ "run_id": "metadata-run-001"
155
+ }
156
+ ```
157
+
158
+ Reference-text mode, CTM output, and unmodified zero-minimum timestamps are mandatory
159
+ and intentionally not configurable.
160
+
161
+ ## WhisperX Configuration
162
+
163
+ Python type: `speech_text_data_aligners.backends.whisperx.config.WhisperXConfig`.
164
+
165
+ | Field | Type | Default | Meaning |
166
+ |---|---|---|---|
167
+ | `python_executable` | path | Current interpreter | Python in the isolated WhisperX runtime |
168
+ | `model_id` | string | `WAV2VEC2_ASR_BASE_960H` | English alignment bundle |
169
+ | `model_revision` | string | `torchaudio-2.8.0` | Model asset revision recorded in provenance |
170
+ | `expected_model_sha256` | string/null | `null` | Optional model-cache aggregate hash |
171
+ | `language` | string | `en` | Alignment language; only `en` is supported |
172
+ | `device` | string | `cpu` | Torch model/alignment device |
173
+ | `model_dir` | path/null | `null` | Persistent model cache directory |
174
+ | `model_cache_only` | boolean | `false` | Disable remote model access where supported |
175
+ | `expected_whisperx_version` | string | `3.8.6` | Required WhisperX distribution |
176
+ | `expected_torch_version` | string | `2.8.0` | Required Torch base version |
177
+ | `expected_torchaudio_version` | string | `2.8.0` | Required Torchaudio base version |
178
+ | `timeout_s` | positive finite number | `3600.0` | Worker-process timeout |
179
+ | `artifact_policy` | enum | `never` | Native artifact retention policy |
180
+ | `temporary_root` | path/null | `null` | Parent for temporary workspaces |
181
+ | `retention_root` | path/null | `null` | Required parent when retention is enabled |
182
+ | `run_id` | string | `whisperx-run` | Safe retained-workspace component |
183
+
184
+ `expected_model_sha256` requires `model_dir`. WhisperX 3.8.6 does not enforce
185
+ `model_cache_only` for Torchaudio bundle downloads, so prepare and hash the cache before
186
+ offline use when remote access must be impossible.
187
+
188
+ CPU example:
189
+
190
+ ```json
191
+ {
192
+ "python_executable": "/opt/conda/envs/speech-aligners-whisperx/bin/python",
193
+ "device": "cpu",
194
+ "model_dir": "/opt/models/whisperx-align",
195
+ "model_cache_only": true,
196
+ "expected_model_sha256": "<lowercase-64-character-sha256>"
197
+ }
198
+ ```
199
+
200
+ The authoritative transcript, full-duration reference segment, disabled interpolation,
201
+ and no-ASR workflow are mandatory and intentionally not configurable.
202
+
203
+ ## Benchmark Composition
204
+
205
+ Pass unique backend names to repeated `--backend` options. The configuration object
206
+ must contain at most those names; entries for unselected names are ignored by the CLI
207
+ factory lookup, while malformed selected values fail construction.
208
+
209
+ ```bash
210
+ speech-text-aligners benchmark \
211
+ metadata.csv first-word-gold.csv wavs \
212
+ --backend mfa --backend whisperx \
213
+ --backend-config-json '{"mfa":{"environment_name":"speech-aligners-mfa"},"whisperx":{"device":"cpu"}}' \
214
+ --output-base reports/mfa-whisperx
215
+ ```
216
+
217
+ For reproducible heterogeneous runs, prefer constructing typed configs in Python and
218
+ calling `benchmark_backends` with an insertion-ordered mapping.
219
+
220
+ ## Readiness Catalog
221
+
222
+ The readiness catalog is a JSON object:
223
+
224
+ ```json
225
+ {
226
+ "backends": {
227
+ "mfa": {
228
+ "identity": {
229
+ "backend_name": "mfa",
230
+ "backend_version": "3.4.2",
231
+ "model_id": "MontrealCorpusTools/english_mfa",
232
+ "model_revision": "1009e3c6c689f10875146a5e4e5c2233f8eaf922"
233
+ },
234
+ "config": {
235
+ "environment_name": "speech-aligners-mfa"
236
+ }
237
+ },
238
+ "whisperx": {
239
+ "identity": {
240
+ "backend_name": "whisperx",
241
+ "backend_version": "3.8.6",
242
+ "model_id": "WAV2VEC2_ASR_BASE_960H"
243
+ },
244
+ "config": null,
245
+ "unavailable_reason": "runtime is intentionally not installed"
246
+ }
247
+ },
248
+ "evidence": [
249
+ {
250
+ "audio_id": "example_audio_id",
251
+ "backend": "mfa",
252
+ "candidates": [
253
+ {
254
+ "path": "/absolute/path/to/evidence.json",
255
+ "sha256": "<lowercase-64-character-sha256>"
256
+ }
257
+ ]
258
+ }
259
+ ]
260
+ }
261
+ ```
262
+
263
+ `identity.backend_name` must match its map key and the constructed adapter name.
264
+ `config: null` means the runtime is explicitly unavailable. Evidence candidates are
265
+ reused only after audio, transcript, backend/model provenance, and complete full-word
266
+ timing are verified. See [W1-T Semantic Readiness](W1_T_SEMANTIC_READINESS.md).
267
+
268
+ ## Provenance Fields
269
+
270
+ For empirical runs, retain at least:
271
+
272
+ - backend and backend package version;
273
+ - model/checkpoint/dictionary identifiers;
274
+ - model revision/hash when feasible;
275
+ - Python and Torch/CUDA versions where relevant;
276
+ - CPU/GPU device;
277
+ - command/configuration identity;
278
+ - input metadata hash;
279
+ - result and diagnostic artifact paths.
280
+
281
+ The runtime receipt and backend result provenance carry these fields. Preserve receipts
282
+ with benchmark outputs; do not infer missing versions from documentation alone.
docs/DOCUMENTATION.md ADDED
@@ -0,0 +1,87 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Documentation Map
2
+
3
+ This repository separates current operating guidance from immutable acceptance and
4
+ empirical history. Start with the README, then use the focused document that matches
5
+ the task.
6
+
7
+ ## Current Guidance
8
+
9
+ | Document | Use it for |
10
+ |---|---|
11
+ | [`README.md`](../README.md) | Hugging Face-style overview, install, quick starts, model matrix, headline metrics |
12
+ | [`config.json`](../config.json) | Machine-readable project, schema, model, and evaluation manifest |
13
+ | [`ARCHITECTURE.md`](../ARCHITECTURE.md) | Current layers, modules, data flow, isolation, and evolution rules |
14
+ | [`CONFIGURATION.md`](CONFIGURATION.md) | Typed fields, defaults, invariants, CLI JSON, readiness catalogs |
15
+ | [`EVALUATION.md`](EVALUATION.md) | Compared configurations, metrics, results, limitations, artifacts |
16
+ | [`WORKFLOWS.md`](WORKFLOWS.md) | End-to-end annotation, benchmark, backend, and readiness flows |
17
+ | [`TESTING.md`](TESTING.md) | Quality commands, markers, coverage, and focused-test map |
18
+ | [`PROJECT_STATUS.md`](PROJECT_STATUS.md) | Accepted operational status and evidence boundary |
19
+ | [`W1_T_SEMANTIC_READINESS.md`](W1_T_SEMANTIC_READINESS.md) | Specialized offline semantic-boundary timing workflow |
20
+
21
+ ## Contracts
22
+
23
+ | Document | Authority |
24
+ |---|---|
25
+ | [`ALIGNER_ENGINEERING_CONTRACT.md`](ALIGNER_ENGINEERING_CONTRACT.md) | Data columns, first/last semantics, reference-text and timestamp invariants |
26
+ | [`MODULAR_ARCHITECTURE_CONTRACT.md`](MODULAR_ARCHITECTURE_CONTRACT.md) | Mandatory layering, dependency direction, reuse, isolation, and coverage |
27
+ | [`AGENTS.md`](../AGENTS.md) | Repository execution and acceptance rules for coding agents |
28
+ | [`SKILLS.md`](../SKILLS.md) | Reusable implementation and verification patterns |
29
+
30
+ Contracts change only when the implementation/API changes intentionally. Update the
31
+ contract and its focused tests together.
32
+
33
+ ## Decision and Empirical Evidence
34
+
35
+ | Document/artifact | Authority |
36
+ |---|---|
37
+ | [`FINAL_ALIGNER_SELECTION.md`](FINAL_ALIGNER_SELECTION.md) | Current operational selection and limitations |
38
+ | [`POSTFINAL_THREE_ALIGNER_COMPARISON.md`](POSTFINAL_THREE_ALIGNER_COMPARISON.md) | Symmetric three-backend full-36 comparison |
39
+ | `results/final/` | Canonical selection and MFA annotated output |
40
+ | `results/postfinal/` | Full-36 MFA extension and three-backend evidence |
41
+ | `results/w1t/` | Readiness timing, consensus, review, and receipt artifacts |
42
+
43
+ Evidence files are preserved as reviewed. Documentation may summarize and link them;
44
+ it must not rewrite hashes, source cells, timing, or historical conclusions.
45
+
46
+ ## Historical Reports
47
+
48
+ | Document | Historical scope |
49
+ |---|---|
50
+ | [`STAGE_0_REPORT.md`](STAGE_0_REPORT.md) | Research, dependency selection, and original architecture freeze |
51
+ | [`STAGE_1_REPORT.md`](STAGE_1_REPORT.md) | Shared framework implementation and closure repairs |
52
+ | [`STAGE_2_REPORT.md`](STAGE_2_REPORT.md) | NeMo adapter implementation and original empirical boundary |
53
+ | [`STAGE_3_REPORT.md`](STAGE_3_REPORT.md) | MFA adapter and original five-row evidence |
54
+ | [`STAGE_4_REPORT.md`](STAGE_4_REPORT.md) | WhisperX adapter and original five-row/full-run evidence |
55
+ | [`STAGE_5_FINAL_ACCEPTANCE_REPORT.md`](STAGE_5_FINAL_ACCEPTANCE_REPORT.md) | Original software/architecture acceptance snapshot |
56
+ | [`POST_STAGE5_DOCS_LICENSE_REPORT.md`](POST_STAGE5_DOCS_LICENSE_REPORT.md) | Documentation, headers, and licensing audit |
57
+
58
+ Stage reports are provenance snapshots. Their counts, paths, and selection statements
59
+ may be superseded by later evidence. Current instructions live in the documents above.
60
+
61
+ ## Maintenance Rules
62
+
63
+ When behavior or evidence changes:
64
+
65
+ 1. Update the current focused guide first.
66
+ 2. Update README/config only for user-facing identity, defaults, models, or headline
67
+ metrics.
68
+ 3. Add or update a result receipt/evidence artifact before citing a new empirical
69
+ claim.
70
+ 4. Keep historical reports unchanged unless correcting a factual link or marking a
71
+ superseding record.
72
+ 5. Validate all local Markdown links and repository paths.
73
+ 6. Run every documented syntax/help/quality command before finalizing.
74
+ 7. Update `.agents/plans`, `.agents/work_data`, and `.agents/logs` with decisions and
75
+ observed results.
76
+ 8. Review `AGENTS.md` and `SKILLS.md` at the end for durable process gaps.
77
+
78
+ ## Terminology
79
+
80
+ - **Authoritative transcript**: the original `en_transcript` cell.
81
+ - **Forced alignment**: locating supplied text in audio; not ASR transcription.
82
+ - **Manual gold**: human reference timing/text fields.
83
+ - **Pairwise disagreement**: difference between two backend outputs; not accuracy.
84
+ - **Structural success**: valid complete word/timing output; not empirical accuracy.
85
+ - **Observed runtime**: one run's measured time under its stated hardware/cache state.
86
+ - **Current guidance**: maintained operating documentation.
87
+ - **Historical report**: preserved stage/acceptance snapshot.
docs/EVALUATION.md ADDED
@@ -0,0 +1,195 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Evaluation
2
+
3
+ This page is the canonical comparison of the alignment models and configurations in
4
+ this repository. It distinguishes manual-gold accuracy, structural success, pairwise
5
+ disagreement, diagnostic behavior, runtime observations, and offline readiness
6
+ agreement. Those evidence types are not interchangeable.
7
+
8
+ ## Evaluation Scope
9
+
10
+ | Evidence set | Rows | Available reference | Purpose |
11
+ |---|---:|---|---|
12
+ | Five-row manual gold | 5 | `first_word_text`, `first_word_end_s` | Limited timing accuracy |
13
+ | Full metadata set | 36 | Authoritative transcripts and WAV bounds | Structural success and cross-backend disagreement |
14
+ | W1-T semantic-readiness timing | 36 | Frozen semantic boundary tokens | Offline automatic END estimates and agreement |
15
+
16
+ The metadata evaluation aligns the supplied `en_transcript`; it does not score ASR
17
+ transcription. The five-row file contains no gold for `first_word_start_s`,
18
+ `last_word_start_s`, or `last_word_end_s`.
19
+
20
+ ## Compared Models and Configurations
21
+
22
+ | Backend | Runtime | Model identity | Dictionary/model revision | Device and cache |
23
+ |---|---|---|---|---|
24
+ | MFA | MFA 3.4.2; Kaldi 5.5.1172; Kalpy 0.10.5; OpenFst 1.8.4 | `MontrealCorpusTools/english_mfa` model 3.3.0 | Revision `1009e3c6c689f10875146a5e4e5c2233f8eaf922`; `english_us_mfa`; G2P enabled | CPU; cold/model-cache conditions vary by run |
25
+ | WhisperX | WhisperX 3.8.6; Torch/Torchaudio 2.8.0 | `WAV2VEC2_ASR_BASE_960H` | Torchaudio 2.8.0 bundle; observed asset SHA-256 `488fd4f...c945d` | CPU; prepopulated hash-verified cache in accepted runs |
26
+ | NeMo / NFA | NeMo 3.0.0 source `fd6a877539710e2b98f28c43272ff81312f83417` | `stt_en_fastconformer_hybrid_large_pc` | Local checkpoint hash `37afe662...2c8b61`; no registry revision recorded | NVIDIA A10G in accepted empirical runs |
27
+
28
+ The full unabbreviated identities and hashes are in [`config.json`](../config.json),
29
+ the run receipts, and [Configuration](CONFIGURATION.md).
30
+
31
+ ## Metric Definitions
32
+
33
+ | Metric | Definition | Missing/failure behavior |
34
+ |---|---|---|
35
+ | Success | A backend returned a complete result that passed lexical/timestamp validation | Failure is counted and never receives invented timing |
36
+ | Exact first-word text | Output `first_word_text` exactly equals the reference surface | Failed rows cannot match |
37
+ | First-word-end absolute error | `abs(predicted_first_word_end_s - reference_first_word_end_s)` | Empty reference is counted as missing and excluded |
38
+ | MAE | Mean of available first-word-end absolute errors | No scorable reference yields null |
39
+ | Median / maximum | Median and maximum available first-word-end absolute error | Same exclusion as MAE |
40
+ | Pairwise boundary difference | Absolute difference between two successful backend boundaries | Only rows successful in both backends |
41
+ | Material disagreement | Any first/last boundary differs by at least the declared threshold | Diagnostic review queue, not an error rate |
42
+ | Runtime | Measured service/batch execution time | Observed only; not normalized across hardware/cache |
43
+
44
+ Benchmark reports expose success/failure counts, missing-reference counts, exact-text
45
+ rate, first-end MAE/median/maximum, runtime, and per-boundary pairwise MAE. Historical
46
+ comparison artifacts additionally calculate pairwise median, p95, maximum, and
47
+ threshold counts.
48
+
49
+ ## Five-Row Manual Gold
50
+
51
+ Evidence date: 2026-09-06 accepted comparison. Sample size: 5.
52
+
53
+ | Backend | Configuration | Success | Exact first text | First-end MAE | Median | Maximum |
54
+ |---|---|---:|---:|---:|---:|---:|
55
+ | MFA | `english_mfa@1009e3c...`, `english_us_mfa` | 5/5 | 5/5 | **0.0162 s** | **0.0110 s** | **0.0300 s** |
56
+ | WhisperX | `WAV2VEC2_ASR_BASE_960H`, CPU | 5/5 | 5/5 | **0.0154 s** | **0.0140 s** | **0.0310 s** |
57
+ | NeMo | `stt_en_fastconformer_hybrid_large_pc`, A10G | 5/5 | 5/5 | **0.2210 s** | **0.0100 s** | **0.9200 s** |
58
+
59
+ Interpretation:
60
+
61
+ - MFA and WhisperX are effectively tied on the only timing gold.
62
+ - WhisperX MAE is lower by 0.0008 s; the sample is too small to rank by this alone.
63
+ - MFA has slightly lower median and maximum error.
64
+ - NeMo has low median error but a 0.920 s maximum on `EN0949`, producing its larger MAE.
65
+ - These results do not establish accuracy for first-word start or either last boundary.
66
+
67
+ ## Full-36 Structural Comparison
68
+
69
+ All three backends produced 36/36 candidates. Every accepted candidate preserved
70
+ source cells/order, the exact six-column extension, authoritative first/last lexical
71
+ surfaces, and finite WAV-bounded monotonic timestamps.
72
+
73
+ ### Pairwise Absolute Differences
74
+
75
+ | Pair | Boundary | Mean | Median | p95 | Maximum |
76
+ |---|---|---:|---:|---:|---:|
77
+ | MFA / WhisperX | first start | 0.059306 s | 0.051000 s | 0.116750 s | 0.132000 s |
78
+ | MFA / WhisperX | first end | 0.018694 s | 0.015000 s | 0.039000 s | 0.078000 s |
79
+ | MFA / WhisperX | last start | 0.211611 s | 0.064000 s | 1.062500 s | 1.545000 s |
80
+ | MFA / WhisperX | last end | 0.325083 s | 0.128000 s | 1.449750 s | 1.835000 s |
81
+ | MFA / NeMo | first start | 0.665556 s | 0.325000 s | 1.430000 s | 2.120000 s |
82
+ | MFA / NeMo | first end | 0.292500 s | 0.170000 s | 1.250000 s | 1.510000 s |
83
+ | MFA / NeMo | last start | 0.137778 s | 0.160000 s | 0.225000 s | 0.250000 s |
84
+ | MFA / NeMo | last end | 0.104444 s | 0.100000 s | 0.222500 s | 0.260000 s |
85
+ | WhisperX / NeMo | first start | 0.724861 s | 0.371500 s | 1.453000 s | 2.122000 s |
86
+ | WhisperX / NeMo | first end | 0.292639 s | 0.151000 s | 1.252250 s | 1.503000 s |
87
+ | WhisperX / NeMo | last start | 0.259500 s | 0.106000 s | 1.247500 s | 1.795000 s |
88
+ | WhisperX / NeMo | last end | 0.398861 s | 0.202000 s | 1.534750 s | 1.955000 s |
89
+
90
+ p95 uses linear interpolation at rank `(n - 1) * 0.95`.
91
+
92
+ ### Material-Disagreement Counts
93
+
94
+ Counts are rows where at least one of the four boundaries reaches the threshold:
95
+
96
+ | Pair | >=0.250 s | >=0.500 s | >=1.000 s |
97
+ |---|---:|---:|---:|
98
+ | MFA / WhisperX | 7 | 6 | 5 |
99
+ | MFA / NeMo | 28 | 17 | 12 |
100
+ | WhisperX / NeMo | 32 | 21 | 16 |
101
+
102
+ MFA and WhisperX are close at first-word boundaries. All seven of their material rows
103
+ are driven by last-word boundaries. These differences identify review cases; they do
104
+ not show which backend is acoustically correct.
105
+
106
+ ## Diagnostic Boundary Signals
107
+
108
+ | Signal | MFA | WhisperX | NeMo |
109
+ |---|---:|---:|---:|
110
+ | Exact zero first-word starts | 0/36 | 0/36 | 17/36 |
111
+ | First-word duration p95 | 0.457 s | 0.400 s | 1.560 s |
112
+ | First-word duration maximum | 0.720 s | 0.621 s | 2.160 s |
113
+ | Final word within 50 ms of WAV end | 0/36 | 0/36 | 0/36 |
114
+
115
+ NeMo's repeated zero starts and the `EN0949` first-end outlier are material operational
116
+ warnings. MFA and NeMo have smaller last-boundary disagreement from each other than
117
+ either has with WhisperX on the largest cases. This is still concordance evidence, not
118
+ manual ground truth.
119
+
120
+ ## Runtime Observations
121
+
122
+ | Backend | Observed run | Runtime | Conditions |
123
+ |---|---|---:|---|
124
+ | MFA | Accepted original five-row cold run | 274.9569 s | Cold run; later run observed 79.5288 s under different cache/host conditions |
125
+ | WhisperX | Accepted five-row run | 28.5392 s | CPU, prepopulated hash-verified model cache |
126
+ | NeMo | Accepted five-row run | 13.7050 s | NVIDIA A10G after smoke; caches may have been warm |
127
+
128
+ Runtime comparisons are secondary. Host, device, process overhead, model download, and
129
+ cache state were not controlled. Do not convert these observations into a general
130
+ speed ranking.
131
+
132
+ ## W1-T Semantic-Readiness Evidence
133
+
134
+ W1-T maps frozen English semantic-boundary tokens to automatic acoustic END estimates.
135
+ It does not measure human Russian onset and does not adjudicate semantic boundaries.
136
+
137
+ | Metric | Observed result |
138
+ |---|---|
139
+ | Cases | 36 |
140
+ | Mapping | 36 EXACT |
141
+ | Confidence | 1 HIGH, 35 MEDIUM |
142
+ | LOW / SINGLE_BACKEND / UNRESOLVED | 0 / 0 / 0 |
143
+ | Cross-backend range p50 | 22.5 ms |
144
+ | Cross-backend range p95 | 86.5 ms |
145
+ | Cross-backend maximum range | 127 ms |
146
+ | Semantic-ready END p50 | 2.1765 s |
147
+ | Semantic-ready END p95 | 3.316625 s |
148
+
149
+ Consensus is the median of valid independent backend END estimates. HIGH requires
150
+ exactly three valid backends and range <=100 ms; MEDIUM requires at least two and range
151
+ <=250 ms; LOW requires at least two and range >250 ms. Agreement is not timing
152
+ accuracy. See [W1-T Semantic Readiness](W1_T_SEMANTIC_READINESS.md) for reuse,
153
+ provenance, mapping, and materialization rules.
154
+
155
+ ## Operational Selection
156
+
157
+ MFA is the selected operational backend as of 2026-09-06. The decision combines:
158
+
159
+ 1. strict 36/36 eligibility;
160
+ 2. a first-word-end MAE effectively tied with WhisperX;
161
+ 3. no repeated zero first-word starts;
162
+ 4. diagnostic last-boundary support on the largest disputed cases;
163
+ 5. reproducible pinned runtime/model identity.
164
+
165
+ This is an operational choice from limited evidence, not a claim of universal timing
166
+ superiority. WhisperX remains the accepted alternative. NeMo remains usable behind the
167
+ same protocol where its pinned GPU runtime/model are available.
168
+
169
+ ## Canonical Artifacts
170
+
171
+ | Artifact | Purpose |
172
+ |---|---|
173
+ | `results/final/final-annotated-36.csv` | Canonical MFA annotation, 36/36 |
174
+ | `results/final/final-annotated-36.receipt.json` | Import/final hashes and evidence links |
175
+ | `results/final/final-aligner-selection.{json,md}` | Canonical selection record |
176
+ | `results/postfinal/three-backend-full-36/` | Full comparison, pairwise tables, runtime and systematic analysis |
177
+ | `results/postfinal/mfa-full-36/` | MFA full-run validation and diagnostics |
178
+ | `results/stage5/` | Historical WhisperX full-run and original acceptance evidence |
179
+ | `results/w1t/` | Offline semantic-readiness sidecars and receipts |
180
+
181
+ The canonical CSV SHA-256 is
182
+ `37bcb85fc22fb142dae40453b90dd3b427bbfd657861bee7529cd433915535a8`.
183
+ Repository CSV copies use LF line endings; import receipts preserve the reviewed
184
+ external artifact hash where line endings differed.
185
+
186
+ ## Limitations
187
+
188
+ - Manual gold covers only first-word text/end on five rows.
189
+ - First-word start and last-word boundaries have no manual ground truth.
190
+ - Pairwise agreement cannot establish acoustic accuracy.
191
+ - The 0.0008 s MFA/WhisperX MAE difference is not decision-grade.
192
+ - Full-corpus pairwise analysis is diagnostic and threshold-dependent.
193
+ - Runtime observations are not hardware/cache controlled.
194
+ - Accents, broader acoustic conditions, repeated runs, and larger human-annotated sets
195
+ remain outside the current evidence boundary.
docs/MODULAR_ARCHITECTURE_CONTRACT.md CHANGED
@@ -32,39 +32,50 @@ src/speech_text_data_aligners/
32
  │ ├── errors.py # project exception taxonomy
33
  │ ├── normalization.py # lexical extraction + reversible mappings
34
  │ ├── validation.py # alignment/timestamp invariants
35
- │ └── extraction.py # first/last lexical boundary selection
 
36
  ├── data/
37
  │ ├── metadata.py # CSV read/write/preservation contract
38
- │ └── audio.py # audio_id/path resolution + lightweight metadata
 
39
  ├── application/
40
  │ ├── annotate.py # reusable annotation use case/service
41
- │ └── benchmark.py # reusable backend evaluation/comparison use case
 
 
42
  ├── benchmarking/
43
  │ ├── metrics.py # deterministic gold/pairwise metrics
44
- │ └── reports.py # machine/human-readable reporting
 
45
  ├── runtime/
46
  │ ├── process.py # injectable subprocess abstraction
47
- │ └── receipts.py # environment/model/run provenance capture
 
48
  ├── backends/
49
  │ ├── nemo/
50
  │ │ ├── backend.py # AlignmentBackend adapter/orchestration
51
  │ │ ├── config.py # typed NFA config
52
  │ │ ├── manifest.py # native input artifact construction
 
53
  │ │ └── ctm.py # native CTM parser/mapping
54
  │ ├── mfa/
55
  │ │ ├── backend.py
56
  │ │ ├── config.py
57
  │ │ ├── corpus.py # temporary corpus/native inputs
 
58
  │ │ └── textgrid.py # word-tier parser/mapping
59
  │ └── whisperx/
60
  │ ├── backend.py
61
  │ ├── config.py
62
  │ ├── provider.py # injectable model/API call boundary
 
 
63
  │ └── parser.py # native result conversion/mapping
64
  └── cli/
65
  ├── main.py # composition root only
66
  ├── annotate.py # argument/presentation adapter
67
- └── benchmark.py # argument/presentation adapter
 
68
  ```
69
 
70
  Do not create files solely to imitate this tree. A smaller module set is acceptable when a file would otherwise be trivial, but every retained module must have one clear responsibility and the layer boundaries below remain mandatory. Conversely, do not collapse unrelated concerns into monolithic backend or CLI modules merely to reduce file count.
@@ -218,3 +229,6 @@ The final report must contain:
218
  6. confirmation that importing the backend-neutral library succeeds without heavyweight optional backends installed;
219
  7. confirmation that each backend can be disabled/absent without breaking the others;
220
  8. any deviations from this contract with explicit rationale.
 
 
 
 
32
  │ ├── errors.py # project exception taxonomy
33
  │ ├── normalization.py # lexical extraction + reversible mappings
34
  │ ├── validation.py # alignment/timestamp invariants
35
+ │ ├── extraction.py # first/last lexical boundary selection
36
+ │ └── readiness.py # semantic-boundary mapping and consensus
37
  ├── data/
38
  │ ├── metadata.py # CSV read/write/preservation contract
39
+ │ ├── audio.py # audio_id/path resolution + lightweight metadata
40
+ │ └── readiness.py # frozen readiness cohort/hash verification
41
  ├── application/
42
  │ ├── annotate.py # reusable annotation use case/service
43
+ │ ├── benchmark.py # reusable backend evaluation/comparison use case
44
+ │ ├── readiness.py # reuse-first offline readiness orchestration
45
+ │ └── materialize.py # accepted-evidence full-word timing materialization
46
  ├── benchmarking/
47
  │ ├── metrics.py # deterministic gold/pairwise metrics
48
+ │ ├── reports.py # machine/human-readable reporting
49
+ │ └── readiness.py # readiness distributions/sensitivity reporting
50
  ├── runtime/
51
  │ ├── process.py # injectable subprocess abstraction
52
+ │ ├── receipts.py # environment/model/run provenance capture
53
+ │ └── alignment_evidence.py # full-word evidence integrity/reuse verification
54
  ├── backends/
55
  │ ├── nemo/
56
  │ │ ├── backend.py # AlignmentBackend adapter/orchestration
57
  │ │ ├── config.py # typed NFA config
58
  │ │ ├── manifest.py # native input artifact construction
59
+ │ │ ├── runner.py # isolated NFA process execution
60
  │ │ └── ctm.py # native CTM parser/mapping
61
  │ ├── mfa/
62
  │ │ ├── backend.py
63
  │ │ ├── config.py
64
  │ │ ├── corpus.py # temporary corpus/native inputs
65
+ │ │ ├── runner.py # isolated MFA process execution
66
  │ │ └── textgrid.py # word-tier parser/mapping
67
  │ └── whisperx/
68
  │ ├── backend.py
69
  │ ├── config.py
70
  │ ├── provider.py # injectable model/API call boundary
71
+ │ ├── worker.py # lazy heavyweight native runtime
72
+ │ ├── segments.py # full-duration authoritative segments
73
  │ └── parser.py # native result conversion/mapping
74
  └── cli/
75
  ├── main.py # composition root only
76
  ├── annotate.py # argument/presentation adapter
77
+ ├── benchmark.py # argument/presentation adapter
78
+ └── readiness.py # readiness catalog/composition adapter
79
  ```
80
 
81
  Do not create files solely to imitate this tree. A smaller module set is acceptable when a file would otherwise be trivial, but every retained module must have one clear responsibility and the layer boundaries below remain mandatory. Conversely, do not collapse unrelated concerns into monolithic backend or CLI modules merely to reduce file count.
 
229
  6. confirmation that importing the backend-neutral library succeeds without heavyweight optional backends installed;
230
  7. confirmation that each backend can be disabled/absent without breaking the others;
231
  8. any deviations from this contract with explicit rationale.
232
+
233
+ The current maintained evidence is summarized in [`ARCHITECTURE.md`](../ARCHITECTURE.md)
234
+ and the implementation-to-test/coverage map in [`TESTING.md`](TESTING.md).
docs/PROJECT_STATUS.md CHANGED
@@ -4,6 +4,11 @@ This page summarizes the accepted Stage 0–5 baseline, NVIDIA-host NeMo closure
4
  post-final three-backend full-36 comparison completed on 2026-09-06. Software acceptance,
5
  empirical validation, and operational selection remain distinct.
6
 
 
 
 
 
 
7
  ## Final Completion
8
 
9
  MFA is the selected operational backend. The canonical reviewed 36-row annotation is
@@ -58,6 +63,21 @@ Manual gold covers only `first_word_text` and `first_word_end_s` on five rows.
58
  `first_word_start_s`, `last_word_start_s`, and `last_word_end_s` do not have manual ground
59
  truth. Pairwise agreement/disagreement is diagnostic evidence, not accuracy.
60
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
61
  ## Detailed Evidence
62
 
63
  - [Stage 0 research and architecture](STAGE_0_REPORT.md)
@@ -68,3 +88,5 @@ truth. Pairwise agreement/disagreement is diagnostic evidence, not accuracy.
68
  - [Stage 5 final acceptance](STAGE_5_FINAL_ACCEPTANCE_REPORT.md)
69
  - [Post-final three-aligner comparison](POSTFINAL_THREE_ALIGNER_COMPARISON.md)
70
  - [Final aligner selection](FINAL_ALIGNER_SELECTION.md)
 
 
 
4
  post-final three-backend full-36 comparison completed on 2026-09-06. Software acceptance,
5
  empirical validation, and operational selection remain distinct.
6
 
7
+ Current operating guidance is in the [README](../README.md),
8
+ [Configuration](CONFIGURATION.md), [Evaluation](EVALUATION.md), [Workflows](WORKFLOWS.md),
9
+ and [Testing and Quality](TESTING.md). Stage reports below are preserved acceptance
10
+ history rather than the primary user guide.
11
+
12
  ## Final Completion
13
 
14
  MFA is the selected operational backend. The canonical reviewed 36-row annotation is
 
63
  `first_word_start_s`, `last_word_start_s`, and `last_word_end_s` do not have manual ground
64
  truth. Pairwise agreement/disagreement is diagnostic evidence, not accuracy.
65
 
66
+ ## Current Engineering Gate
67
+
68
+ The documentation-review checkout on 2026-10-30 passes the deterministic suite and
69
+ quality tooling. This is a software-quality result, not additional acoustic evidence.
70
+
71
+ | Gate | Current result |
72
+ |---|---|
73
+ | Deterministic tests | 635 passed, 3 heavyweight integration tests skipped |
74
+ | Aggregate line coverage | 98.74% |
75
+ | Individual implementation coverage | All pass; minimum 93.33% |
76
+ | Ruff / strict mypy | Pass |
77
+ | Import Linter | 7 contracts kept |
78
+
79
+ See [Testing and Quality](TESTING.md) for commands and the complete file/test map.
80
+
81
  ## Detailed Evidence
82
 
83
  - [Stage 0 research and architecture](STAGE_0_REPORT.md)
 
88
  - [Stage 5 final acceptance](STAGE_5_FINAL_ACCEPTANCE_REPORT.md)
89
  - [Post-final three-aligner comparison](POSTFINAL_THREE_ALIGNER_COMPARISON.md)
90
  - [Final aligner selection](FINAL_ALIGNER_SELECTION.md)
91
+ - [Canonical evaluation summary](EVALUATION.md)
92
+ - [Configuration reference](CONFIGURATION.md)
docs/TESTING.md ADDED
@@ -0,0 +1,177 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Testing and Quality
2
+
3
+ This page defines the deterministic release gate and records the current per-file
4
+ coverage map. Ordinary tests do not download models, call remote services, require a
5
+ GPU, or invoke heavyweight aligners.
6
+
7
+ ## Environment
8
+
9
+ Use the core development environment or the local equivalent:
10
+
11
+ ```bash
12
+ conda activate speech-text-data-aligners
13
+ python -m pip install -e . --no-deps
14
+ ```
15
+
16
+ Pinned development tools are in [`requirements.txt`](../requirements.txt) and
17
+ [`environments/core-dev.yml`](../environments/core-dev.yml).
18
+
19
+ ## Deterministic Gate
20
+
21
+ Run from the repository root:
22
+
23
+ ```bash
24
+ python -m pytest
25
+ ruff check .
26
+ ruff format --check .
27
+ python -m mypy src
28
+ lint-imports
29
+ coverage erase
30
+ coverage run -m pytest
31
+ coverage report --show-missing
32
+ coverage json -o coverage.json
33
+ python -m pytest tests/quality/test_per_file_coverage.py -q
34
+ ```
35
+
36
+ `coverage.json` is generated evidence and is ignored by Git. Regenerate it before
37
+ running the per-file gate; a stale file is not accepted as current proof.
38
+
39
+ ## Current Observed Gate
40
+
41
+ Captured in this checkout on 2026-10-30 before documentation-only changes to source:
42
+
43
+ | Gate | Result |
44
+ |---|---|
45
+ | Pytest | 635 passed, 3 skipped |
46
+ | Aggregate implementation line coverage | 98.74% |
47
+ | Lowest implementation file | 93.33% |
48
+ | Lowest shared-layer file | 95.65% |
49
+ | Ruff lint/format | Pass |
50
+ | Strict mypy | Pass in 53 source files |
51
+ | Import Linter | 7 kept, 0 broken |
52
+ | Architecture tests | 10 passed |
53
+ | Header/manifest/per-file tests | Pass |
54
+
55
+ The repository gate is at least 90% for every non-empty implementation file and at
56
+ least 90% aggregate. This repository additionally enforces at least 95% for shared
57
+ `core`, `data`, `application`, `benchmarking`, and `runtime` files.
58
+
59
+ ## Test Boundaries
60
+
61
+ | Test area | Purpose |
62
+ |---|---|
63
+ | `tests/unit/core` | Normalization, mapping, models, validation, errors, extraction, readiness |
64
+ | `tests/unit/data` | CSV preservation, WAV resolution/probing, readiness cohort verification |
65
+ | `tests/unit/application` | Public use cases, publication, benchmarking, readiness, materialization |
66
+ | `tests/unit/benchmarking` | Metrics, reports, readiness distributions |
67
+ | `tests/unit/runtime` | Process, artifacts, receipts, evidence integrity |
68
+ | `tests/unit/backends` | Configs, native builders/parsers, adapters, registry, isolated providers |
69
+ | `tests/unit/cli` | Argument validation and delegation to application services |
70
+ | `tests/architecture` | Dependency direction and readiness isolation |
71
+ | `tests/quality` | Headers, manifest consistency, per-file coverage |
72
+ | `tests/integration/*` | Opt-in real subprocess/model smoke tests |
73
+
74
+ Integration tests are marked `integration` plus `mfa`, `nemo`, or `whisperx`. They are
75
+ excluded from ordinary confidence unless their runtime and explicit inputs are
76
+ available.
77
+
78
+ ## Opt-In Integration
79
+
80
+ MFA:
81
+
82
+ ```bash
83
+ RUN_MFA_INTEGRATION=1 \
84
+ MFA_CONDA_PREFIX=/opt/conda/envs/speech-aligners-mfa \
85
+ MFA_SMOKE_WAV=/absolute/path/to/file.wav \
86
+ MFA_SMOKE_TRANSCRIPT='Exact authoritative transcript.' \
87
+ python -m pytest tests/integration/mfa/test_mfa_real.py -m 'integration and mfa' -v
88
+ ```
89
+
90
+ WhisperX:
91
+
92
+ ```bash
93
+ RUN_WHISPERX_INTEGRATION=1 \
94
+ WHISPERX_PYTHON=/opt/conda/envs/speech-aligners-whisperx/bin/python \
95
+ WHISPERX_SMOKE_WAV=/absolute/path/to/file.wav \
96
+ WHISPERX_SMOKE_TRANSCRIPT='Exact authoritative transcript.' \
97
+ WHISPERX_MODEL_DIR=/opt/models/whisperx-align \
98
+ python -m pytest tests/integration/whisperx/test_whisperx_real.py -m 'integration and whisperx' -v
99
+ ```
100
+
101
+ NeMo:
102
+
103
+ ```bash
104
+ NEMO_PYTHON=/opt/runtimes/nemo-speech-v3.0.0/.venv/bin/python \
105
+ NEMO_ALIGN_SCRIPT=/opt/runtimes/nemo-speech-v3.0.0/tools/nemo_forced_aligner/align.py \
106
+ NEMO_MODEL_PATH=/opt/models/stt_en_fastconformer_hybrid_large_pc.nemo \
107
+ NEMO_WAV_PATH=/absolute/path/to/file.wav \
108
+ NEMO_TRANSCRIPT='Exact authoritative transcript.' \
109
+ NEMO_DEVICE=cuda \
110
+ python -m pytest tests/integration/nemo/test_nemo_real.py -m 'integration and nemo' -v
111
+ ```
112
+
113
+ NeMo may download the registry model only after `NEMO_ALLOW_MODEL_DOWNLOAD=1` is set
114
+ explicitly. A local checkpoint is preferred. Integration commands require the listed
115
+ hardware/data and are not part of ordinary unit CI.
116
+
117
+ ## Implementation-to-Test and Coverage Map
118
+
119
+ Coverage values are current line coverage for each implementation file. Empty package
120
+ markers are intentionally zero bytes and excluded from implementation coverage; they
121
+ provide no executable behavior.
122
+
123
+ | Implementation | Focused test | Coverage |
124
+ |---|---|---:|
125
+ | `speech_text_data_aligners/__init__.py` | `tests/unit/test_public_api.py` | 100.00% |
126
+ | `application/annotate.py` | `tests/unit/application/test_annotate.py` | 98.00% |
127
+ | `application/benchmark.py` | `tests/unit/application/test_benchmark.py` | 100.00% |
128
+ | `application/materialize.py` | `tests/unit/application/test_materialize.py` | 95.68% |
129
+ | `application/readiness.py` | `tests/unit/application/test_readiness.py` | 100.00% |
130
+ | `backends/registry.py` | `tests/unit/backends/test_registry.py` | 100.00% |
131
+ | `backends/mfa/backend.py` | `tests/unit/backends/mfa/test_backend.py` | 100.00% |
132
+ | `backends/mfa/config.py` | `tests/unit/backends/mfa/test_config.py` | 99.03% |
133
+ | `backends/mfa/corpus.py` | `tests/unit/backends/mfa/test_corpus.py` | 99.19% |
134
+ | `backends/mfa/runner.py` | `tests/unit/backends/mfa/test_runner.py` | 98.62% |
135
+ | `backends/mfa/textgrid.py` | `tests/unit/backends/mfa/test_textgrid.py` | 97.91% |
136
+ | `backends/nemo/backend.py` | `tests/unit/backends/nemo/test_backend.py` | 100.00% |
137
+ | `backends/nemo/config.py` | `tests/unit/backends/nemo/test_config.py` | 98.11% |
138
+ | `backends/nemo/ctm.py` | `tests/unit/backends/nemo/test_ctm.py` | 100.00% |
139
+ | `backends/nemo/manifest.py` | `tests/unit/backends/nemo/test_manifest.py` | 100.00% |
140
+ | `backends/nemo/runner.py` | `tests/unit/backends/nemo/test_runner.py` | 100.00% |
141
+ | `backends/whisperx/backend.py` | `tests/unit/backends/whisperx/test_backend.py` | 100.00% |
142
+ | `backends/whisperx/config.py` | `tests/unit/backends/whisperx/test_config.py` | 100.00% |
143
+ | `backends/whisperx/parser.py` | `tests/unit/backends/whisperx/test_parser.py` | 100.00% |
144
+ | `backends/whisperx/provider.py` | `tests/unit/backends/whisperx/test_provider.py` | 98.74% |
145
+ | `backends/whisperx/segments.py` | `tests/unit/backends/whisperx/test_segments.py` | 100.00% |
146
+ | `backends/whisperx/worker.py` | `tests/unit/backends/whisperx/test_worker.py` | 93.33% |
147
+ | `benchmarking/metrics.py` | `tests/unit/benchmarking/test_metrics.py` | 100.00% |
148
+ | `benchmarking/readiness.py` | `tests/unit/benchmarking/test_readiness.py` | 100.00% |
149
+ | `benchmarking/reports.py` | `tests/unit/benchmarking/test_reports.py` | 98.84% |
150
+ | `cli/annotate.py` | `tests/unit/cli/test_annotate.py` | 100.00% |
151
+ | `cli/benchmark.py` | `tests/unit/cli/test_benchmark.py` | 100.00% |
152
+ | `cli/main.py` | `tests/unit/cli/test_main.py` | 95.83% |
153
+ | `cli/readiness.py` | `tests/unit/cli/test_readiness.py` | 100.00% |
154
+ | `core/errors.py` | `tests/unit/core/test_errors.py` | 100.00% |
155
+ | `core/extraction.py` | `tests/unit/core/test_extraction.py` | 95.65% |
156
+ | `core/models.py` | `tests/unit/core/test_models.py` | 100.00% |
157
+ | `core/normalization.py` | `tests/unit/core/test_normalization.py` | 100.00% |
158
+ | `core/protocols.py` | `tests/unit/core/test_protocols.py` | 100.00% |
159
+ | `core/readiness.py` | `tests/unit/core/test_readiness.py` | 100.00% |
160
+ | `core/validation.py` | `tests/unit/core/test_validation.py` | 98.33% |
161
+ | `data/audio.py` | `tests/unit/data/test_audio.py` | 97.37% |
162
+ | `data/metadata.py` | `tests/unit/data/test_metadata.py` | 98.08% |
163
+ | `data/readiness.py` | `tests/unit/data/test_readiness.py` | 100.00% |
164
+ | `runtime/alignment_evidence.py` | `tests/unit/runtime/test_alignment_evidence.py` | 100.00% |
165
+ | `runtime/artifacts.py` | `tests/unit/runtime/test_artifacts.py` | 100.00% |
166
+ | `runtime/process.py` | `tests/unit/runtime/test_process.py` | 100.00% |
167
+ | `runtime/receipts.py` | `tests/unit/runtime/test_receipts.py` | 100.00% |
168
+
169
+ ## Coverage Interpretation
170
+
171
+ - Coverage is line coverage from the deterministic full suite with real integration
172
+ tests skipped.
173
+ - High coverage does not replace behavioral review or empirical alignment evidence.
174
+ - A skipped integration test is not counted as a successful real backend run.
175
+ - Aggregate coverage cannot compensate for a file below its individual threshold.
176
+ - When an implementation file is added, add its focused test and update this map in the
177
+ same change.
docs/W1_T_SEMANTIC_READINESS.md CHANGED
@@ -1,5 +1,9 @@
1
  # W1-T semantic-readiness timing
2
 
 
 
 
 
3
  W1-T maps an already frozen source-English semantic word boundary to an automatic
4
  acoustic END estimate. It does not adjudicate the semantic boundary or measure human
5
  Russian onset. Every sidecar record fixes `human_measured=false`,
 
1
  # W1-T semantic-readiness timing
2
 
3
+ This document preserves the specialized W1-T workflow and its 2026-09-10 evidence
4
+ snapshot. For the current repository-wide quality command sequence and current test
5
+ counts, use [Testing and Quality](TESTING.md).
6
+
7
  W1-T maps an already frozen source-English semantic word boundary to an automatic
8
  acoustic END estimate. It does not adjudicate the semantic boundary or measure human
9
  Russian onset. Every sidecar record fixes `human_measured=false`,
docs/WORKFLOWS.md CHANGED
@@ -116,6 +116,39 @@ accuracy.
116
  The Python benchmark service returns a typed report without writing. The CLI can print
117
  one selected format or preflight and publish a deterministic JSON/CSV/Markdown bundle.
118
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
119
  ## NeMo Workflow
120
 
121
  ```text
@@ -259,3 +292,41 @@ required result/reference values exist.
259
  For API examples and the concise entry point, return to the
260
  [repository README](../README.md). For the current empirical boundary, see
261
  [Project Status](PROJECT_STATUS.md).
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
116
  The Python benchmark service returns a typed report without writing. The CLI can print
117
  one selected format or preflight and publish a deterministic JSON/CSV/Markdown bundle.
118
 
119
+ ## Offline Readiness Workflow
120
+
121
+ ```text
122
+ exact three-column metadata + frozen boundaries + exact WAV directory
123
+ |
124
+ v
125
+ validate cohort size, unique audio IDs, transcript/audio hashes, and boundaries
126
+ |
127
+ v
128
+ audit supplied full-word evidence capsules
129
+ +-- verified case/backend --> reuse
130
+ `-- missing/rejected ------> provider execution
131
+ |
132
+ +-- unavailable runtime --> structured status
133
+ +-- typed failure --------> structured status
134
+ `-- successful result ----> verify and save capsule
135
+ |
136
+ v
137
+ map frozen semantic token to original lexical span
138
+ |
139
+ v
140
+ extract mapped word END values and compute median/range/MAD
141
+ |
142
+ v
143
+ timing sidecar + summary + review queue + receipt + reuse ledger
144
+ ```
145
+
146
+ The readiness CLI composes providers from an explicit catalog. It runs only missing or
147
+ rejected case/backend pairs and refuses to overwrite an existing output directory.
148
+ The service marks its sidecar `evaluation_only=true`, `human_measured=false`, and
149
+ `runtime_usable=false`. See [Configuration](CONFIGURATION.md) for catalog structure and
150
+ [W1-T Semantic Readiness](W1_T_SEMANTIC_READINESS.md) for mapping/consensus semantics.
151
+
152
  ## NeMo Workflow
153
 
154
  ```text
 
292
  For API examples and the concise entry point, return to the
293
  [repository README](../README.md). For the current empirical boundary, see
294
  [Project Status](PROJECT_STATUS.md).
295
+
296
+ ## Command and Artifact Map
297
+
298
+ | Command | Primary inputs | Primary outputs | Exit behavior |
299
+ |---|---|---|---|
300
+ | `annotate` | Metadata CSV, WAV directory, backend config | Annotated CSV; optional diagnostics/receipt | `0` success; typed project errors exit `2` |
301
+ | `benchmark` | Metadata, sparse reference, WAV directory, one or more backend configs | Rendered report or JSON/CSV/Markdown bundle | `0` after report; typed project errors exit `2` |
302
+ | `readiness` | Three-column metadata, frozen boundaries, WAV directory, catalog | Four primary W1-T JSON/JSONL artifacts plus reuse/attempt evidence | `0` timing-complete; `3` timing-blocked; typed errors exit `2` |
303
+
304
+ Annotation outputs:
305
+
306
+ | Artifact | When written | Contract |
307
+ |---|---|---|
308
+ | Annotated CSV | Every completed continue-on-error run and every successful strict run | Source preserved; exactly six columns appended |
309
+ | Diagnostics JSON | When one or more rows fail | Ordered structured failures; no fabricated timing |
310
+ | Receipt JSON | When requested | Input hash, backend, counts, host, timing, artifact paths |
311
+
312
+ Benchmark outputs:
313
+
314
+ | Artifact | Contract |
315
+ |---|---|
316
+ | JSON | Complete typed observations, gold metrics, pairwise metrics |
317
+ | CSV | Deterministic `scope,metric,value` headline rows |
318
+ | Markdown | Human-readable headline metrics |
319
+
320
+ Readiness outputs in a new attempt directory:
321
+
322
+ | Artifact | Contract |
323
+ |---|---|
324
+ | `W1_T_SEMANTIC_READINESS_TIMING.jsonl` | One ordered row per frozen case |
325
+ | `W1_T_SUMMARY.json` | Distribution, confidence, and acceptance summary |
326
+ | `W1_T_REVIEW_QUEUE.jsonl` | Failures, provenance gaps, ambiguity, and low confidence |
327
+ | `W1_T_RECEIPT.json` | Cohort hashes, host, timing gate, and status |
328
+ | `reuse_ledger*.jsonl` | Pre-run and final evidence reuse decisions |
329
+ | `backend_runs/` | Requests and new canonical result/failure evidence |
330
+
331
+ Configuration fields and exact JSON shapes are in [Configuration](CONFIGURATION.md).
332
+ Metric interpretation is in [Evaluation](EVALUATION.md).
tests/quality/test_config_manifest.py ADDED
@@ -0,0 +1,76 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # License: Apache-2.0 License
2
+ # Created by: Patrick Lumbantobing, VertoX-AI
3
+ # Copyright (c) 2026 VertoX-AI. All rights reserved.
4
+ #
5
+ # This work is licensed under the Apache-2.0 License.
6
+ # To view a copy of this license, visit
7
+ # https://www.apache.org/licenses/LICENSE-2.0
8
+
9
+ """Validate the machine-readable project manifest against stable source contracts."""
10
+
11
+ from __future__ import annotations
12
+
13
+ import hashlib
14
+ import json
15
+ import tomllib
16
+ from pathlib import Path
17
+
18
+ from speech_text_data_aligners.backends.mfa.config import (
19
+ DEFAULT_MODEL_ID as MFA_MODEL_ID,
20
+ )
21
+ from speech_text_data_aligners.backends.mfa.config import (
22
+ DEFAULT_MODEL_REVISION as MFA_REVISION,
23
+ )
24
+ from speech_text_data_aligners.backends.nemo.config import (
25
+ DEFAULT_MODEL_ID as NEMO_MODEL_ID,
26
+ )
27
+ from speech_text_data_aligners.backends.registry import default_backend_registry
28
+ from speech_text_data_aligners.backends.whisperx.config import (
29
+ DEFAULT_MODEL_ID as WHISPERX_MODEL_ID,
30
+ )
31
+ from speech_text_data_aligners.core.extraction import BOUNDARY_COLUMNS
32
+ from speech_text_data_aligners.data.metadata import SOURCE_COLUMNS
33
+
34
+
35
+ def _manifest() -> dict[str, object]:
36
+ """Load the root manifest as a JSON object.
37
+
38
+ Returns
39
+ -------
40
+ Parsed manifest values.
41
+
42
+ """
43
+ return json.loads(Path("config.json").read_text(encoding="utf-8"))
44
+
45
+
46
+ def test_manifest_matches_registry_schema_and_canonical_values() -> None:
47
+ """Assert that the manifest cannot drift from registry/schema/model defaults."""
48
+ manifest = _manifest()
49
+ assert manifest["schema_version"] == "speech-text-data-aligners/config.v1"
50
+ data_contract = manifest["data_contract"]
51
+ assert data_contract["source_columns"] == list(SOURCE_COLUMNS)
52
+ assert data_contract["alignment_columns"] == list(BOUNDARY_COLUMNS)
53
+ registry = manifest["backend_registry"]
54
+ assert registry["operational_selection"] == "mfa"
55
+ assert set(registry["models"]) == set(default_backend_registry().names)
56
+ models = registry["models"]
57
+ assert models["mfa"]["model"]["id"] == MFA_MODEL_ID
58
+ assert models["mfa"]["model"]["revision"] == MFA_REVISION
59
+ assert models["nemo"]["model"]["id"] == NEMO_MODEL_ID
60
+ assert models["whisperx"]["model"]["id"] == WHISPERX_MODEL_ID
61
+ evaluation = manifest["evaluation"]
62
+ assert evaluation["manual_gold"]["rows"] == 5
63
+ assert evaluation["manual_gold"]["fields"] == [
64
+ "first_word_text",
65
+ "first_word_end_s",
66
+ ]
67
+ assert (
68
+ evaluation["canonical_output"]["path"] == "results/final/final-annotated-36.csv"
69
+ )
70
+ project_config = tomllib.loads(Path("pyproject.toml").read_text(encoding="utf-8"))
71
+ assert manifest["project"]["version"] == project_config["project"]["version"]
72
+ canonical = Path(evaluation["canonical_output"]["path"]).read_bytes()
73
+ assert (
74
+ evaluation["canonical_output"]["sha256"]
75
+ == hashlib.sha256(canonical).hexdigest()
76
+ )
tests/quality/test_documentation_links.py ADDED
@@ -0,0 +1,67 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # License: Apache-2.0 License
2
+ # Created by: Patrick Lumbantobing, VertoX-AI
3
+ # Copyright (c) 2026 VertoX-AI. All rights reserved.
4
+ #
5
+ # This work is licensed under the Apache-2.0 License.
6
+ # To view a copy of this license, visit
7
+ # https://www.apache.org/licenses/LICENSE-2.0
8
+
9
+ """Verify that repository Markdown documents resolve their local link targets."""
10
+
11
+ from __future__ import annotations
12
+
13
+ import re
14
+ from pathlib import Path
15
+ from urllib.parse import unquote, urlsplit
16
+
17
+ PROJECT_ROOT = Path(__file__).resolve().parents[2]
18
+ MARKDOWN_LINK = re.compile(r"(?<!\!)\[[^\]]*\]\(([^)\s]+)(?:\s+[^)]*)?\)")
19
+ SKIP_PREFIXES = ("http://", "https://", "mailto:", "data:", "#")
20
+
21
+
22
+ def _markdown_files() -> tuple[Path, ...]:
23
+ """Return maintained repository Markdown files in deterministic order.
24
+
25
+ Returns
26
+ -------
27
+ Root and tracked-area Markdown paths excluding generated/agent-private trees.
28
+
29
+ """
30
+ return tuple(
31
+ sorted(
32
+ path
33
+ for path in PROJECT_ROOT.rglob("*.md")
34
+ if not any(
35
+ part
36
+ in {".agents", ".git", ".mypy_cache", ".pytest_cache", ".ruff_cache"}
37
+ for part in path.relative_to(PROJECT_ROOT).parts
38
+ )
39
+ )
40
+ )
41
+
42
+
43
+ def test_local_markdown_links_resolve() -> None:
44
+ """Fail when any local Markdown link targets a missing path."""
45
+ failures: list[str] = []
46
+ for document in _markdown_files():
47
+ content = document.read_text(encoding="utf-8")
48
+ for raw_target in MARKDOWN_LINK.findall(content):
49
+ target = unquote(urlsplit(raw_target).path)
50
+ if not target or raw_target.startswith(SKIP_PREFIXES):
51
+ continue
52
+ resolved = (
53
+ Path(target) if target.startswith("/") else document.parent / target
54
+ ).resolve()
55
+ try:
56
+ resolved.relative_to(PROJECT_ROOT.resolve())
57
+ except ValueError:
58
+ failures.append(
59
+ f"{document.relative_to(PROJECT_ROOT)}: link escapes repository: "
60
+ f"{raw_target}"
61
+ )
62
+ continue
63
+ if not resolved.exists():
64
+ failures.append(
65
+ f"{document.relative_to(PROJECT_ROOT)}: missing target: {raw_target}"
66
+ )
67
+ assert not failures, "broken local Markdown links:\n" + "\n".join(failures)