src/docs/TESTING.md

# Tests

Every comparison is bitwise (arrays through their raw bytes, files byte for byte). Property tests are
derandomised (hypothesis profile "det", fixed seeds), so every run checks the same cases. All suites run in the
canonical environment through `make` (`env/canonical.sh`, `python -s`, the project venv).

| Command | Needs | What it checks |
|---|---|---|
| `make test-rust` | CPU | Rust unit tests: kernels, plan edges, span windows vs direct windows, median, pathlib and repr rules |
| `make test-fast` | CPU | numpy/scipy/Python differential tests of every exact kernel; audio decoding against soundfile over every format, and 2,750 damaged files (seeded byte flips, cuts, edge-value header words, insertions, junk) decoded and loaded exactly as soundfile and the release do, numpy's allocation-error text; Python `repr` of every Unicode code point; front-end plan and batches against pyannote's own code; masks against the release; the engine with stand-in networks against the release's own `analyse` (records, RTTMs, errors, batch shapes), cancellation, per-file errors, BLAS environment independence; 40 cached files through the post-processing; VBx row-parallel products vs OpenBLAS's one-call product; the gateway as a service in front of a stand-in Triton (`tests/test_gateway.py`, 32 tests: every RTTM, error and record against the release with the same stand-in networks under each of the three schedules, with one Triton per GPU, and with 64 clients; the fair and share schedules ordering work as documented; Triton killed and restarted mid-run; a model that stays away; model errors; a crashed model process; the gateway killed and restarted mid-run, the client's job ids still working after their jobs were submitted again; queue and upload limits; stalled uploads; a sample rate that would resample past the size limit; names; path rules; the token; the client refusing redirects; cancellation, also when a synchronous client leaves; the result cache; metrics; the spool; SIGTERM; 2,000 seeded random requests (methods, paths, query strings, headers, bodies) without a server error) |
| `make test-cache` | CPU | the identity check on cached records (`tests/test_identity_cached.py`): all 1,945 cached records' RTTMs; every replay input decoded; float16 rounding over all 2^32 float32 inputs |
| `make test-gpu` | GPUs | the identity check on audio, sample (`tests/test_identity_audio.py`: about 240 files: records, RTTMs, errors) and the GPU tests of `tests/test_api.py` (below) |
| `make test-gpu-all` | GPUs | the identity check on audio over all 1,973 files (291.6 h) and the determinism matrix (below) |
| `triton/identity_check.sh` | GPUs, Docker | the identity check through the serving stack (`tests/test_identity_serving.py`): Triton + gateway + client, concurrent shuffled jobs, RTTMs, errors and dumped records |
| `triton/identity_check.sh "gpu and slow"` | GPUs, Docker | the same over all 1,973 files |
| `python -s -m pytest tests/test_api.py -m gpu` | GPUs | `Diarizer(...)(file)`, `Diarizer.run(..., out_dir)`, the `anuvani-diar` CLI vs the release's golden hashes and `run.json` fields, and the install check |
| `anuvani-diar-check --audio-dir DIR [--long-dir artifacts/test_audio]` | GPUs | the install check: the release's 12 golden fingerprints, plus the 1 h, 3 h and 6 h files; `tests/test_api.py::test_install_check` runs it |
| `python -s -m pytest tests/test_matrix.py -m gpu` | GPUs | the determinism matrix: 39 files on each GPU alone and on all of them together; post-processing threads 1/2/8/64, decoding threads 1/2/4/8, files at once 1/8/64/default; a new arrival order per row; a repeat in the same warm process; a fresh process with every BLAS and OpenMP thread variable at 64. Records and RTTMs vs the reference outputs: identical in every run |
| `GW_ARGS=... IDENTITY_SEED=... RESTART_AT=... triton/identity_check.sh` | GPUs, Docker | variants of the serving identity check: another GPU assignment and schedule; Triton killed and restarted mid-run |

The identity checks compare with the reference program's outputs, and hold on the setup the release's outputs were
produced with (`env/canonical.sh`); on other GPUs or library versions only the network-free parts (`make test-fast`,
`make test-cache`) are expected to match byte for byte.

Seeded, repeatable equivalents take the place of fuzzing (fuzzing is random from run to run, and cargo-fuzz needs a
nightly toolchain): 2,750 damaged audio files through the decoder
wrapper, and 2,000 random requests through the job API.

Reference outputs the suites read: `oracle/cache_canonical.json` and `artifacts/oracle_cache/` (the identity check
on cached records), `oracle/audio_canonical.json` and `artifacts/oracle_audio/` (the identity checks on audio and
through the serving stack), `assets/nn_golden.json` (the network self-test). `oracle/` and `artifacts/` are not part
of the repository: they are the reference program's (`reference/` in the model repository) outputs on the benchmark,
and the suites read them, with the benchmark manifests (`manifests/`) and the cached front-end records
(`records/<set>/`), from `$ANUVANI_DIAR_DEV_DATA`, by default `artifacts/dev_data/`.