Anuvani-Diarization / src /docs /DETERMINISM.md
StephennFernandes's picture
Anuvani-Diarization 1.0.0
63dd946
|
Raw History Blame Contribute Delete
4.84 kB

Determinism: why the outputs are byte-identical, and where that stops

The engine reproduces the release's arithmetic, not just its results. Each piece below is pinned for a measured reason; docs/TESTING.md lists the checks that hold it in place, and tests/test_matrix.py runs the same 39 files under changing conditions (each GPU alone and all of them together, 1 to 64 threads, 1 to 64 files at once, other arrival orders, a warm repeat, a fresh process with every BLAS thread variable at 64): every record and RTTM identical in all of them.

The networks (GPU)

  • Same code and weights. The release's own model classes (DiariZen and its pyannote fork at commit 844f555), the bundle's weights, built the way load_bundle builds them.
  • Same numeric flags, set after the imports. FP32, TF32 off, deterministic algorithms, cuDNN benchmark off, CUBLAS_WORKSPACE_CONFIG=:4096:8. Importing DiariZen switches TF32 on for the whole process, so the flags are set after it, as load_bundle does, and re-asserted before every network call.
  • Same kernels. torch 2.1.1+cu121, its cuDNN 8.9.2, and cuBLAS 12.1.0.26 (env/canonical.sh loads it from a CUDA 12.1 installation; the cu121 Triton image carries the same files and searches them first).
  • Same batch shapes. Segmentation: batches of 8, the remainder, the zero-padded last window alone (other sizes change bits). Voiceprints: the trunk once per window in batches of 8, pooling and the last layer at the release's rows (identical bits), the odd last window as the release runs it.
  • One CUDA stream per GPU. Every GPU of a kind gives the same bits as the others, but two streams sharing one model on one GPU changed about 1 batch in 1,000. Locally there is one worker thread per GPU; on Triton one process per GPU.
  • Models never cross GPUs. Moving several model copies from GPU to GPU in one process gave all-NaN weights on one GPU; copies go through host memory, and every Triton instance sees only its own GPU.
  • Checked before use. Every backend runs every batch shape at start-up and compares with the release's own outputs (assets/nn_golden.json, written from load_bundle): identical bytes where the setup is reproduced; otherwise the outputs must stay within fixed near-identical bounds of assets/nn_reference.npz (or, in strict mode, the backend stops).

Everything else (CPU, Rust)

  • numpy, scipy, pyannote and CPython reproduced operation by operation (crate diar-exact): pairwise summation with numpy's 8 accumulators and 8,192-element buffers, numpy's BLAS dispatch (gemm / gemv / dot) with numpy's own OpenBLAS binary at one thread, glibc's exp/log, introsort tie order, scipy's centroid linkage and heap, cosine distance with its two-lane sum, rectangular assignment, pyannote's sliding-window arithmetic and aggregation, CPython's round, sum, string sort and .3f formatting.
  • Parallel only where each element's arithmetic is unchanged. Distances by row, VBx products by blocks of result rows (each its own single-threaded OpenBLAS call; tested bitwise against the one-call product), VBx and aggregation by rows or output frames with every element's terms in the release's order.
  • The engine's OpenBLAS is private (its own linker namespace) and pinned to one thread whatever the environment says.
  • Decoding is libsndfile 1.2.2, the binary soundfile loads, called in soundfile's order; non-16 kHz audio is resampled by the release's own torchaudio resampler (copied unchanged into anuvani_diar.nn.resample) with 2 threads.

Where the guarantee stops

  • Identical bytes need the same software (env/canonical.sh) on the same kind of GPU as the release's outputs were produced with. Another GPU model, or another PyTorch, CUDA or cuDNN version, picks other kernels whose last bits differ: the start-up check then measures the networks against the release's outputs and runs only if they stay within the near-identical bounds (maximum posterior difference 0.25, frame decisions at least 99% equal, voiceprint cosine at least 0.995), and says so; strict mode refuses to run instead.
  • The CPU side is pinned to AVX2 code paths by env/canonical.sh (numpy, ATen and oneDNN features, and OPENBLAS_CORETYPE=Zen). The engine's private OpenBLAS no longer depends on that variable: it sets OPENBLAS_CORETYPE=Zen (and one thread) itself for its own load when the CPU reports AVX2 and FMA, checks that it runs core Zen, and refuses to run otherwise (docs/WHEEL.md). On an AVX2 CPU the pins change nothing; on a CPU with AVX-512, running without them could select other kernels in numpy, ATen and oneDNN.
  • Inputs the release itself would treat differently (a different bundle, batch size or package versions) are outside the contract.