| # Testing and Quality |
|
|
| This page defines the deterministic release gate and records the current per-file |
| coverage map. Ordinary tests do not download models, call remote services, require a |
| GPU, or invoke heavyweight aligners. |
|
|
| ## Environment |
|
|
| Use the core development environment or the local equivalent: |
|
|
| ```bash |
| conda activate speech-text-data-aligners |
| python -m pip install -e . --no-deps |
| ``` |
|
|
| Pinned development tools are in [`requirements.txt`](../requirements.txt) and |
| [`environments/core-dev.yml`](../environments/core-dev.yml). |
|
|
| ## Deterministic Gate |
|
|
| Run from the repository root: |
|
|
| ```bash |
| python -m pytest |
| ruff check . |
| ruff format --check . |
| python -m mypy src |
| lint-imports |
| coverage erase |
| coverage run -m pytest |
| coverage report --show-missing |
| coverage json -o coverage.json |
| python -m pytest tests/quality/test_per_file_coverage.py -q |
| ``` |
|
|
| `coverage.json` is generated evidence and is ignored by Git. Regenerate it before |
| running the per-file gate; a stale file is not accepted as current proof. |
|
|
| ## Current Observed Gate |
|
|
| Captured in this checkout on 2026-10-30 before documentation-only changes to source: |
|
|
| | Gate | Result | |
| |---|---| |
| | Pytest | 635 passed, 3 skipped | |
| | Aggregate implementation line coverage | 98.74% | |
| | Lowest implementation file | 93.33% | |
| | Lowest shared-layer file | 95.65% | |
| | Ruff lint/format | Pass | |
| | Strict mypy | Pass in 53 source files | |
| | Import Linter | 7 kept, 0 broken | |
| | Architecture tests | 10 passed | |
| | Header/manifest/per-file tests | Pass | |
|
|
| The repository gate is at least 90% for every non-empty implementation file and at |
| least 90% aggregate. This repository additionally enforces at least 95% for shared |
| `core`, `data`, `application`, `benchmarking`, and `runtime` files. |
|
|
| ## Test Boundaries |
|
|
| | Test area | Purpose | |
| |---|---| |
| | `tests/unit/core` | Normalization, mapping, models, validation, errors, extraction, readiness | |
| | `tests/unit/data` | CSV preservation, WAV resolution/probing, readiness cohort verification | |
| | `tests/unit/application` | Public use cases, publication, benchmarking, readiness, materialization | |
| | `tests/unit/benchmarking` | Metrics, reports, readiness distributions | |
| | `tests/unit/runtime` | Process, artifacts, receipts, evidence integrity | |
| | `tests/unit/backends` | Configs, native builders/parsers, adapters, registry, isolated providers | |
| | `tests/unit/cli` | Argument validation and delegation to application services | |
| | `tests/architecture` | Dependency direction and readiness isolation | |
| | `tests/quality` | Headers, manifest consistency, per-file coverage | |
| | `tests/integration/*` | Opt-in real subprocess/model smoke tests | |
|
|
| Integration tests are marked `integration` plus `mfa`, `nemo`, or `whisperx`. They are |
| excluded from ordinary confidence unless their runtime and explicit inputs are |
| available. |
|
|
| ## Opt-In Integration |
|
|
| MFA: |
|
|
| ```bash |
| RUN_MFA_INTEGRATION=1 \ |
| MFA_CONDA_PREFIX=/opt/conda/envs/speech-aligners-mfa \ |
| MFA_SMOKE_WAV=/absolute/path/to/file.wav \ |
| MFA_SMOKE_TRANSCRIPT='Exact authoritative transcript.' \ |
| python -m pytest tests/integration/mfa/test_mfa_real.py -m 'integration and mfa' -v |
| ``` |
|
|
| WhisperX: |
|
|
| ```bash |
| RUN_WHISPERX_INTEGRATION=1 \ |
| WHISPERX_PYTHON=/opt/conda/envs/speech-aligners-whisperx/bin/python \ |
| WHISPERX_SMOKE_WAV=/absolute/path/to/file.wav \ |
| WHISPERX_SMOKE_TRANSCRIPT='Exact authoritative transcript.' \ |
| WHISPERX_MODEL_DIR=/opt/models/whisperx-align \ |
| python -m pytest tests/integration/whisperx/test_whisperx_real.py -m 'integration and whisperx' -v |
| ``` |
|
|
| NeMo: |
|
|
| ```bash |
| NEMO_PYTHON=/opt/runtimes/nemo-speech-v3.0.0/.venv/bin/python \ |
| NEMO_ALIGN_SCRIPT=/opt/runtimes/nemo-speech-v3.0.0/tools/nemo_forced_aligner/align.py \ |
| NEMO_MODEL_PATH=/opt/models/stt_en_fastconformer_hybrid_large_pc.nemo \ |
| NEMO_WAV_PATH=/absolute/path/to/file.wav \ |
| NEMO_TRANSCRIPT='Exact authoritative transcript.' \ |
| NEMO_DEVICE=cuda \ |
| python -m pytest tests/integration/nemo/test_nemo_real.py -m 'integration and nemo' -v |
| ``` |
|
|
| NeMo may download the registry model only after `NEMO_ALLOW_MODEL_DOWNLOAD=1` is set |
| explicitly. A local checkpoint is preferred. Integration commands require the listed |
| hardware/data and are not part of ordinary unit CI. |
|
|
| ## Implementation-to-Test and Coverage Map |
|
|
| Coverage values are current line coverage for each implementation file. Empty package |
| markers are intentionally zero bytes and excluded from implementation coverage; they |
| provide no executable behavior. |
|
|
| | Implementation | Focused test | Coverage | |
| |---|---|---:| |
| | `speech_text_data_aligners/__init__.py` | `tests/unit/test_public_api.py` | 100.00% | |
| | `application/annotate.py` | `tests/unit/application/test_annotate.py` | 98.00% | |
| | `application/benchmark.py` | `tests/unit/application/test_benchmark.py` | 100.00% | |
| | `application/materialize.py` | `tests/unit/application/test_materialize.py` | 95.68% | |
| | `application/readiness.py` | `tests/unit/application/test_readiness.py` | 100.00% | |
| | `backends/registry.py` | `tests/unit/backends/test_registry.py` | 100.00% | |
| | `backends/mfa/backend.py` | `tests/unit/backends/mfa/test_backend.py` | 100.00% | |
| | `backends/mfa/config.py` | `tests/unit/backends/mfa/test_config.py` | 99.03% | |
| | `backends/mfa/corpus.py` | `tests/unit/backends/mfa/test_corpus.py` | 99.19% | |
| | `backends/mfa/runner.py` | `tests/unit/backends/mfa/test_runner.py` | 98.62% | |
| | `backends/mfa/textgrid.py` | `tests/unit/backends/mfa/test_textgrid.py` | 97.91% | |
| | `backends/nemo/backend.py` | `tests/unit/backends/nemo/test_backend.py` | 100.00% | |
| | `backends/nemo/config.py` | `tests/unit/backends/nemo/test_config.py` | 98.11% | |
| | `backends/nemo/ctm.py` | `tests/unit/backends/nemo/test_ctm.py` | 100.00% | |
| | `backends/nemo/manifest.py` | `tests/unit/backends/nemo/test_manifest.py` | 100.00% | |
| | `backends/nemo/runner.py` | `tests/unit/backends/nemo/test_runner.py` | 100.00% | |
| | `backends/whisperx/backend.py` | `tests/unit/backends/whisperx/test_backend.py` | 100.00% | |
| | `backends/whisperx/config.py` | `tests/unit/backends/whisperx/test_config.py` | 100.00% | |
| | `backends/whisperx/parser.py` | `tests/unit/backends/whisperx/test_parser.py` | 100.00% | |
| | `backends/whisperx/provider.py` | `tests/unit/backends/whisperx/test_provider.py` | 98.74% | |
| | `backends/whisperx/segments.py` | `tests/unit/backends/whisperx/test_segments.py` | 100.00% | |
| | `backends/whisperx/worker.py` | `tests/unit/backends/whisperx/test_worker.py` | 93.33% | |
| | `benchmarking/metrics.py` | `tests/unit/benchmarking/test_metrics.py` | 100.00% | |
| | `benchmarking/readiness.py` | `tests/unit/benchmarking/test_readiness.py` | 100.00% | |
| | `benchmarking/reports.py` | `tests/unit/benchmarking/test_reports.py` | 98.84% | |
| | `cli/annotate.py` | `tests/unit/cli/test_annotate.py` | 100.00% | |
| | `cli/benchmark.py` | `tests/unit/cli/test_benchmark.py` | 100.00% | |
| | `cli/main.py` | `tests/unit/cli/test_main.py` | 95.83% | |
| | `cli/readiness.py` | `tests/unit/cli/test_readiness.py` | 100.00% | |
| | `core/errors.py` | `tests/unit/core/test_errors.py` | 100.00% | |
| | `core/extraction.py` | `tests/unit/core/test_extraction.py` | 95.65% | |
| | `core/models.py` | `tests/unit/core/test_models.py` | 100.00% | |
| | `core/normalization.py` | `tests/unit/core/test_normalization.py` | 100.00% | |
| | `core/protocols.py` | `tests/unit/core/test_protocols.py` | 100.00% | |
| | `core/readiness.py` | `tests/unit/core/test_readiness.py` | 100.00% | |
| | `core/validation.py` | `tests/unit/core/test_validation.py` | 98.33% | |
| | `data/audio.py` | `tests/unit/data/test_audio.py` | 97.37% | |
| | `data/metadata.py` | `tests/unit/data/test_metadata.py` | 98.08% | |
| | `data/readiness.py` | `tests/unit/data/test_readiness.py` | 100.00% | |
| | `runtime/alignment_evidence.py` | `tests/unit/runtime/test_alignment_evidence.py` | 100.00% | |
| | `runtime/artifacts.py` | `tests/unit/runtime/test_artifacts.py` | 100.00% | |
| | `runtime/process.py` | `tests/unit/runtime/test_process.py` | 100.00% | |
| | `runtime/receipts.py` | `tests/unit/runtime/test_receipts.py` | 100.00% | |
|
|
| ## Coverage Interpretation |
|
|
| - Coverage is line coverage from the deterministic full suite with real integration |
| tests skipped. |
| - High coverage does not replace behavioral review or empirical alignment evidence. |
| - A skipped integration test is not counted as a successful real backend run. |
| - Aggregate coverage cannot compensate for a file below its individual threshold. |
| - When an implementation file is added, add its focused test and update this map in the |
| same change. |
|
|
