Blame for docs/CONFIGURATION.md

# Configuration

This page is the authoritative configuration reference for Speech-Text Data Aligners.
Typed backend dataclasses validate configuration before model or process execution.
Unknown keys and invalid primitive types are rejected rather than ignored.

## Manifest Versus Runtime Configuration

[`config.json`](../config.json) is a versioned machine-readable project/model manifest.
It records package identity, the data contract, registered backends, pinned model
identity, evaluation scope, and documentation paths. It is validated by
`tests/quality/test_config_manifest.py`.

The manifest is not passed directly to `--backend-config-json`. Runtime configuration
has these CLI shapes:

| Command | `--backend-config-json` shape |
|---|---|
| `annotate` | One selected backend's configuration object, for example `{"environment_name":"speech-aligners-mfa"}` |
| `benchmark` | One outer object keyed by each selected backend, for example `{"mfa":{...},"whisperx":{...}}` |
| `readiness` | Provider configuration lives under each catalog `backends.<name>.config` value |

All path-valued JSON strings are converted to `pathlib.Path`. `null` is accepted only
for fields documented as optional or nullable.

## Environment Isolation

| Runtime | Specification | Default selector | Notes |
|---|---|---|---|
| Core development | `environments/core-dev.yml` | `speech-aligners-core-dev` | Deterministic tests, lint, type, coverage, and import checks |
| MFA | `environments/mfa.yml` | `speech-aligners-mfa` | Dedicated CPU Conda runtime |
| NeMo | `environments/nemo.yml` | No default environment name | Requires explicit pinned source/script and local or registry model |
| WhisperX | `environments/whisperx.yml` | Current Python by default | Dedicated CPU runtime is recommended |

MFA and NeMo execute through an isolated process boundary. WhisperX executes in a
dedicated worker that imports Torch/Torchaudio/WhisperX only after invocation. Heavy
backend packages are never imported by core, data, benchmarking, or application layers.

## Common Artifact Policy

MFA, NeMo, and WhisperX share these artifact semantics:

| `artifact_policy` | Behavior |
|---|---|
| `never` | Remove adapter workspaces; default |
| `on_failure` | Retain native workspace only after failure |
| `always` | Retain native workspace on success or failure |

When the policy is not `never`, `retention_root` is required. `run_id` must be one safe
path component. `temporary_root` is an optional parent for unique workspaces.

Retained native artifacts are diagnostic evidence. They do not replace the
backend-neutral result, and native CTM/TextGrid/WhisperX structures do not cross the
adapter interface.

## MFA Configuration

Python type: `speech_text_data_aligners.backends.mfa.config.MfaConfig`.

| Field | Type | Default | Meaning |
|---|---|---|---|
| `conda_executable` | string | `conda` | Conda command or executable path |
| `environment_name` | string/null | `speech-aligners-mfa` | Dedicated environment name |
| `environment_prefix` | path/null | `null` | Dedicated environment prefix |
| `mfa_executable` | string | `mfa` | Executable inside the selected environment |
| `model_id` | string | `MontrealCorpusTools/english_mfa` | Hugging Face MFA model repository, without revision |
| `model_revision` | string | `1009e3c6c689f10875146a5e4e5c2233f8eaf922` | Required 40-character lowercase commit |
| `model_version` | string | `3.3.0` | Model bundle version recorded in provenance |
| `dialect` | string | `english_us_mfa` | Dictionary/G2P dialect |
| `use_g2p` | boolean | `true` | Generate pronunciations for dictionary OOVs |
| `expected_mfa_version` | string | `3.4.2` | Required MFA distribution |
| `expected_kaldi_version` | string | `5.5.1172` | Required Kaldi distribution |
| `expected_kalpy_version` | string | `0.10.5` | Required Kalpy distribution |
| `expected_openfst_version` | string | `1.8.4` | Required OpenFst distribution |
| `expected_model_sha256` | string/null | `null` | Optional lowercase model-tree SHA-256 |
| `num_jobs` | positive integer | `1` | MFA worker process count |
| `timeout_s` | positive finite number | `3600.0` | Timeout for probe and alignment commands |
| `artifact_policy` | enum | `never` | Native artifact retention policy |
| `temporary_root` | path/null | `null` | Parent for temporary workspaces |
| `retention_root` | path/null | `null` | Required parent when retention is enabled |
| `run_id` | string | `mfa-run` | Safe retained-workspace component |

Exactly one of `environment_name` and `environment_prefix` must be set. When selecting
a prefix from CLI JSON, use `"environment_name": null` explicitly. The adapter always
uses long TextGrid output and a commit-qualified model; those safety choices are not
configurable.

Minimal CLI JSON:

```json
{
  "environment_name": "speech-aligners-mfa",
  "num_jobs": 1,
  "artifact_policy": "never"
}
```

Failure-only artifact example:

```json
{
  "environment_prefix": "/opt/conda/envs/speech-aligners-mfa",
  "environment_name": null,
  "artifact_policy": "on_failure",
  "retention_root": "artifacts/mfa",
  "run_id": "metadata-run-001"
}
```

## NeMo / NFA Configuration

Python type: `speech_text_data_aligners.backends.nemo.config.NemoConfig`.

| Field | Type | Default | Meaning |
|---|---|---|---|
| `align_script` | path | Required | Exact `tools/nemo_forced_aligner/align.py` |
| `python_executable` | path | Current interpreter | Python in the isolated NeMo runtime |
| `pretrained_name` | string/null | `stt_en_fastconformer_hybrid_large_pc` | Registry model name |
| `model_path` | path/null | `null` | Local `.nemo` checkpoint |
| `model_id` | string | `stt_en_fastconformer_hybrid_large_pc` | Stable model identity for provenance |
| `model_revision` | string/null | `null` | Registry/source revision when known |
| `expected_model_sha256` | string/null | `null` | Required local checkpoint SHA-256 |
| `expected_nemo_version` | string | `3.0.0` | Required NeMo distribution |
| `nemo_source_revision` | string | `fd6a877539710e2b98f28c43272ff81312f83417` | Pinned Speech checkout |
| `expected_align_script_sha256` | string | `1cfaf9526b47037204b7874c4432e4d5d07364a7fca1b840052c370d46ee2f03` | Required entry-point hash |
| `transcribe_device` | string | `cuda` | Torch device for acoustic inference |
| `viterbi_device` | string | `cuda` | Torch device for Viterbi alignment |
| `batch_size` | positive integer | `1` | NFA batch size |
| `use_local_attention` | boolean | `true` | Request supported local attention |
| `timeout_s` | positive finite number | `3600.0` | Timeout for probe and alignment |
| `save_ass` | boolean | `false` | Retain optional ASS output in addition to CTM |
| `artifact_policy` | enum | `never` | Native artifact retention policy |
| `temporary_root` | path/null | `null` | Parent for temporary workspaces |
| `retention_root` | path/null | `null` | Required parent when retention is enabled |
| `run_id` | string | `nemo-run` | Safe retained-workspace component |

Exactly one of `pretrained_name` and `model_path` must be set. When using a local
checkpoint from CLI JSON, set `"pretrained_name": null` explicitly.
`expected_model_sha256` is valid only with `model_path`.

Local-checkpoint example:

```json
{
  "align_script": "/opt/runtimes/nemo-speech-v3.0.0/tools/nemo_forced_aligner/align.py",
  "python_executable": "/opt/runtimes/nemo-speech-v3.0.0/.venv/bin/python",
  "pretrained_name": null,
  "model_path": "/opt/models/stt_en_fastconformer_hybrid_large_pc.nemo",
  "expected_model_sha256": "<lowercase-64-character-sha256>",
  "transcribe_device": "cuda",
  "viterbi_device": "cuda",
  "artifact_policy": "on_failure",
  "retention_root": "artifacts/nemo",
  "run_id": "metadata-run-001"
}
```

Reference-text mode, CTM output, and unmodified zero-minimum timestamps are mandatory
and intentionally not configurable.

## WhisperX Configuration

Python type: `speech_text_data_aligners.backends.whisperx.config.WhisperXConfig`.

| Field | Type | Default | Meaning |
|---|---|---|---|
| `python_executable` | path | Current interpreter | Python in the isolated WhisperX runtime |
| `model_id` | string | `WAV2VEC2_ASR_BASE_960H` | English alignment bundle |
| `model_revision` | string | `torchaudio-2.8.0` | Model asset revision recorded in provenance |
| `expected_model_sha256` | string/null | `null` | Optional model-cache aggregate hash |
| `language` | string | `en` | Alignment language; only `en` is supported |
| `device` | string | `cpu` | Torch model/alignment device |
| `model_dir` | path/null | `null` | Persistent model cache directory |
| `model_cache_only` | boolean | `false` | Disable remote model access where supported |
| `expected_whisperx_version` | string | `3.8.6` | Required WhisperX distribution |
| `expected_torch_version` | string | `2.8.0` | Required Torch base version |
| `expected_torchaudio_version` | string | `2.8.0` | Required Torchaudio base version |
| `timeout_s` | positive finite number | `3600.0` | Worker-process timeout |
| `artifact_policy` | enum | `never` | Native artifact retention policy |
| `temporary_root` | path/null | `null` | Parent for temporary workspaces |
| `retention_root` | path/null | `null` | Required parent when retention is enabled |
| `run_id` | string | `whisperx-run` | Safe retained-workspace component |

`expected_model_sha256` requires `model_dir`. WhisperX 3.8.6 does not enforce
`model_cache_only` for Torchaudio bundle downloads, so prepare and hash the cache before
offline use when remote access must be impossible.

CPU example:

```json
{
  "python_executable": "/opt/conda/envs/speech-aligners-whisperx/bin/python",
  "device": "cpu",
  "model_dir": "/opt/models/whisperx-align",
  "model_cache_only": true,
  "expected_model_sha256": "<lowercase-64-character-sha256>"
}
```

The authoritative transcript, full-duration reference segment, disabled interpolation,
and no-ASR workflow are mandatory and intentionally not configurable.

## Benchmark Composition

Pass unique backend names to repeated `--backend` options. The configuration object
must contain at most those names; entries for unselected names are ignored by the CLI
factory lookup, while malformed selected values fail construction.

```bash
speech-text-aligners benchmark \
  metadata.csv first-word-gold.csv wavs \
  --backend mfa --backend whisperx \
  --backend-config-json '{"mfa":{"environment_name":"speech-aligners-mfa"},"whisperx":{"device":"cpu"}}' \
  --output-base reports/mfa-whisperx
```

For reproducible heterogeneous runs, prefer constructing typed configs in Python and
calling `benchmark_backends` with an insertion-ordered mapping.

## Readiness Catalog

The readiness catalog is a JSON object:

```json
{
  "backends": {
    "mfa": {
      "identity": {
        "backend_name": "mfa",
        "backend_version": "3.4.2",
        "model_id": "MontrealCorpusTools/english_mfa",
        "model_revision": "1009e3c6c689f10875146a5e4e5c2233f8eaf922"
      },
      "config": {
        "environment_name": "speech-aligners-mfa"
      }
    },
    "whisperx": {
      "identity": {
        "backend_name": "whisperx",
        "backend_version": "3.8.6",
        "model_id": "WAV2VEC2_ASR_BASE_960H"
      },
      "config": null,
      "unavailable_reason": "runtime is intentionally not installed"
    }
  },
  "evidence": [
    {
      "audio_id": "example_audio_id",
      "backend": "mfa",
      "candidates": [
        {
          "path": "/absolute/path/to/evidence.json",
          "sha256": "<lowercase-64-character-sha256>"
        }
      ]
    }
  ]
}
```

`identity.backend_name` must match its map key and the constructed adapter name.
`config: null` means the runtime is explicitly unavailable. Evidence candidates are
reused only after audio, transcript, backend/model provenance, and complete full-word
timing are verified. See [W1-T Semantic Readiness](W1_T_SEMANTIC_READINESS.md).

## Provenance Fields

For empirical runs, retain at least:

- backend and backend package version;
- model/checkpoint/dictionary identifiers;
- model revision/hash when feasible;
- Python and Torch/CUDA versions where relevant;
- CPU/GPU device;
- command/configuration identity;
- input metadata hash;
- result and diagnostic artifact paths.

The runtime receipt and backend result provenance carry these fields. Preserve receipts
with benchmark outputs; do not infer missing versions from documentation alone.