File size: 8,397 Bytes
a3b146e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 | # Testing and Quality
This page defines the deterministic release gate and records the current per-file
coverage map. Ordinary tests do not download models, call remote services, require a
GPU, or invoke heavyweight aligners.
## Environment
Use the core development environment or the local equivalent:
```bash
conda activate speech-text-data-aligners
python -m pip install -e . --no-deps
```
Pinned development tools are in [`requirements.txt`](../requirements.txt) and
[`environments/core-dev.yml`](../environments/core-dev.yml).
## Deterministic Gate
Run from the repository root:
```bash
python -m pytest
ruff check .
ruff format --check .
python -m mypy src
lint-imports
coverage erase
coverage run -m pytest
coverage report --show-missing
coverage json -o coverage.json
python -m pytest tests/quality/test_per_file_coverage.py -q
```
`coverage.json` is generated evidence and is ignored by Git. Regenerate it before
running the per-file gate; a stale file is not accepted as current proof.
## Current Observed Gate
Captured in this checkout on 2026-10-30 before documentation-only changes to source:
| Gate | Result |
|---|---|
| Pytest | 635 passed, 3 skipped |
| Aggregate implementation line coverage | 98.74% |
| Lowest implementation file | 93.33% |
| Lowest shared-layer file | 95.65% |
| Ruff lint/format | Pass |
| Strict mypy | Pass in 53 source files |
| Import Linter | 7 kept, 0 broken |
| Architecture tests | 10 passed |
| Header/manifest/per-file tests | Pass |
The repository gate is at least 90% for every non-empty implementation file and at
least 90% aggregate. This repository additionally enforces at least 95% for shared
`core`, `data`, `application`, `benchmarking`, and `runtime` files.
## Test Boundaries
| Test area | Purpose |
|---|---|
| `tests/unit/core` | Normalization, mapping, models, validation, errors, extraction, readiness |
| `tests/unit/data` | CSV preservation, WAV resolution/probing, readiness cohort verification |
| `tests/unit/application` | Public use cases, publication, benchmarking, readiness, materialization |
| `tests/unit/benchmarking` | Metrics, reports, readiness distributions |
| `tests/unit/runtime` | Process, artifacts, receipts, evidence integrity |
| `tests/unit/backends` | Configs, native builders/parsers, adapters, registry, isolated providers |
| `tests/unit/cli` | Argument validation and delegation to application services |
| `tests/architecture` | Dependency direction and readiness isolation |
| `tests/quality` | Headers, manifest consistency, per-file coverage |
| `tests/integration/*` | Opt-in real subprocess/model smoke tests |
Integration tests are marked `integration` plus `mfa`, `nemo`, or `whisperx`. They are
excluded from ordinary confidence unless their runtime and explicit inputs are
available.
## Opt-In Integration
MFA:
```bash
RUN_MFA_INTEGRATION=1 \
MFA_CONDA_PREFIX=/opt/conda/envs/speech-aligners-mfa \
MFA_SMOKE_WAV=/absolute/path/to/file.wav \
MFA_SMOKE_TRANSCRIPT='Exact authoritative transcript.' \
python -m pytest tests/integration/mfa/test_mfa_real.py -m 'integration and mfa' -v
```
WhisperX:
```bash
RUN_WHISPERX_INTEGRATION=1 \
WHISPERX_PYTHON=/opt/conda/envs/speech-aligners-whisperx/bin/python \
WHISPERX_SMOKE_WAV=/absolute/path/to/file.wav \
WHISPERX_SMOKE_TRANSCRIPT='Exact authoritative transcript.' \
WHISPERX_MODEL_DIR=/opt/models/whisperx-align \
python -m pytest tests/integration/whisperx/test_whisperx_real.py -m 'integration and whisperx' -v
```
NeMo:
```bash
NEMO_PYTHON=/opt/runtimes/nemo-speech-v3.0.0/.venv/bin/python \
NEMO_ALIGN_SCRIPT=/opt/runtimes/nemo-speech-v3.0.0/tools/nemo_forced_aligner/align.py \
NEMO_MODEL_PATH=/opt/models/stt_en_fastconformer_hybrid_large_pc.nemo \
NEMO_WAV_PATH=/absolute/path/to/file.wav \
NEMO_TRANSCRIPT='Exact authoritative transcript.' \
NEMO_DEVICE=cuda \
python -m pytest tests/integration/nemo/test_nemo_real.py -m 'integration and nemo' -v
```
NeMo may download the registry model only after `NEMO_ALLOW_MODEL_DOWNLOAD=1` is set
explicitly. A local checkpoint is preferred. Integration commands require the listed
hardware/data and are not part of ordinary unit CI.
## Implementation-to-Test and Coverage Map
Coverage values are current line coverage for each implementation file. Empty package
markers are intentionally zero bytes and excluded from implementation coverage; they
provide no executable behavior.
| Implementation | Focused test | Coverage |
|---|---|---:|
| `speech_text_data_aligners/__init__.py` | `tests/unit/test_public_api.py` | 100.00% |
| `application/annotate.py` | `tests/unit/application/test_annotate.py` | 98.00% |
| `application/benchmark.py` | `tests/unit/application/test_benchmark.py` | 100.00% |
| `application/materialize.py` | `tests/unit/application/test_materialize.py` | 95.68% |
| `application/readiness.py` | `tests/unit/application/test_readiness.py` | 100.00% |
| `backends/registry.py` | `tests/unit/backends/test_registry.py` | 100.00% |
| `backends/mfa/backend.py` | `tests/unit/backends/mfa/test_backend.py` | 100.00% |
| `backends/mfa/config.py` | `tests/unit/backends/mfa/test_config.py` | 99.03% |
| `backends/mfa/corpus.py` | `tests/unit/backends/mfa/test_corpus.py` | 99.19% |
| `backends/mfa/runner.py` | `tests/unit/backends/mfa/test_runner.py` | 98.62% |
| `backends/mfa/textgrid.py` | `tests/unit/backends/mfa/test_textgrid.py` | 97.91% |
| `backends/nemo/backend.py` | `tests/unit/backends/nemo/test_backend.py` | 100.00% |
| `backends/nemo/config.py` | `tests/unit/backends/nemo/test_config.py` | 98.11% |
| `backends/nemo/ctm.py` | `tests/unit/backends/nemo/test_ctm.py` | 100.00% |
| `backends/nemo/manifest.py` | `tests/unit/backends/nemo/test_manifest.py` | 100.00% |
| `backends/nemo/runner.py` | `tests/unit/backends/nemo/test_runner.py` | 100.00% |
| `backends/whisperx/backend.py` | `tests/unit/backends/whisperx/test_backend.py` | 100.00% |
| `backends/whisperx/config.py` | `tests/unit/backends/whisperx/test_config.py` | 100.00% |
| `backends/whisperx/parser.py` | `tests/unit/backends/whisperx/test_parser.py` | 100.00% |
| `backends/whisperx/provider.py` | `tests/unit/backends/whisperx/test_provider.py` | 98.74% |
| `backends/whisperx/segments.py` | `tests/unit/backends/whisperx/test_segments.py` | 100.00% |
| `backends/whisperx/worker.py` | `tests/unit/backends/whisperx/test_worker.py` | 93.33% |
| `benchmarking/metrics.py` | `tests/unit/benchmarking/test_metrics.py` | 100.00% |
| `benchmarking/readiness.py` | `tests/unit/benchmarking/test_readiness.py` | 100.00% |
| `benchmarking/reports.py` | `tests/unit/benchmarking/test_reports.py` | 98.84% |
| `cli/annotate.py` | `tests/unit/cli/test_annotate.py` | 100.00% |
| `cli/benchmark.py` | `tests/unit/cli/test_benchmark.py` | 100.00% |
| `cli/main.py` | `tests/unit/cli/test_main.py` | 95.83% |
| `cli/readiness.py` | `tests/unit/cli/test_readiness.py` | 100.00% |
| `core/errors.py` | `tests/unit/core/test_errors.py` | 100.00% |
| `core/extraction.py` | `tests/unit/core/test_extraction.py` | 95.65% |
| `core/models.py` | `tests/unit/core/test_models.py` | 100.00% |
| `core/normalization.py` | `tests/unit/core/test_normalization.py` | 100.00% |
| `core/protocols.py` | `tests/unit/core/test_protocols.py` | 100.00% |
| `core/readiness.py` | `tests/unit/core/test_readiness.py` | 100.00% |
| `core/validation.py` | `tests/unit/core/test_validation.py` | 98.33% |
| `data/audio.py` | `tests/unit/data/test_audio.py` | 97.37% |
| `data/metadata.py` | `tests/unit/data/test_metadata.py` | 98.08% |
| `data/readiness.py` | `tests/unit/data/test_readiness.py` | 100.00% |
| `runtime/alignment_evidence.py` | `tests/unit/runtime/test_alignment_evidence.py` | 100.00% |
| `runtime/artifacts.py` | `tests/unit/runtime/test_artifacts.py` | 100.00% |
| `runtime/process.py` | `tests/unit/runtime/test_process.py` | 100.00% |
| `runtime/receipts.py` | `tests/unit/runtime/test_receipts.py` | 100.00% |
## Coverage Interpretation
- Coverage is line coverage from the deterministic full suite with real integration
tests skipped.
- High coverage does not replace behavioral review or empirical alignment evidence.
- A skipped integration test is not counted as a successful real backend run.
- Aggregate coverage cannot compensate for a file below its individual threshold.
- When an implementation file is added, add its focused test and update this map in the
same change.
|