| # The wheel: anuvani_diar as one self-contained manylinux package |
| |
| `tools/build_wheel.sh` builds `anuvani_diar-<version>-cp310-abi3-manylinux_2_28_x86_64.whl`: one wheel for CPython |
| 3.10 and later (abi3) on x86_64 Linux with glibc 2.28 or newer. It carries the Rust extension, the Python package, the |
| four native libraries the engine loads at run time, the licence texts and the clustering assets, so the CPU side |
| needs nothing from the build machine. |
|
|
| Release 1.0.0 was built on 10 October 2026 from commit 2ce08e60 (clean tree): 14,910,918 bytes, 245 files (189 of them |
| licence files in `.dist-info/licenses/`), sha256 `84d7b5e71eec083a3835f0cc3340f846f3e03562cf2ed6f054476f4107fdd3b1`; |
| the extension `_core.abi3.so` is 2,571,976 bytes after stripping. The first build (9 October, commit 7c648240, 36 |
| files) was 13,657,026 bytes. Each build writes `<wheel>.build.json` (inputs, versions, checks) and each check |
| `<wheel>.check.json` beside the wheel. |
|
|
| ## What is bundled |
|
|
| `anuvani_diar/_libs/`: unmodified copies of the binaries the release's own Python stack uses, sha256-checked when staged, |
| in the built wheel, and again by the engine before it loads them. |
|
|
| | File | What, from where | sha256 | Licence (text in `_libs/LICENSES/`) | |
| |---|---|---|---| |
| | `libopenblas64_p-r0-0cf96a72.3.23.dev.so` | OpenBLAS 0.3.23.dev, ILP64, with LAPACK; numpy 1.26.4 wheel (`numpy.libs/`) | `9254d085…` | BSD-3-Clause (LAPACK: BSD-3-Clause-Attribution); `numpy-1.26.4-LICENSE.txt` | |
| | `libgfortran-040039e1.so.5.0.0` | GCC Fortran runtime; same wheel | `14afb312…` | GPL-3.0 with the GCC Runtime Library Exception 3.1; same file (both full texts) | |
| | `libquadmath-96973f99.so.0.0.0` | GCC quad-precision maths; same wheel | `934c22de…` | LGPL-2.1-or-later; same file, full text in `libsndfile-1.2.2-COPYING.txt` | |
| | `libsndfile_x86_64.so` | libsndfile 1.2.2 with libogg 1.3, libvorbis 1.3.7, FLAC 1.4.3, opus 1.4, mpg123 1.32.3, LAME 3.100 linked in; soundfile 0.14.0 wheel (`_soundfile_data/`) | `c872fb84…` | LGPL-2.1 (`libsndfile-1.2.2-COPYING.txt`, soundfile's copy, identical to libsndfile's own); codecs: BSD-3-Clause (ogg, vorbis, FLAC, opus), LGPL-2.1 (mpg123), LGPL-2.0-or-later (LAME), one file each | |
|
|
| `_libs/LICENSES/README.txt` says which file covers which library and where each comes from. The numpy and soundfile |
| texts are copied from the reference environment; the codec texts (`tools/wheel_licences/`) were taken once from each |
| project's source at the version in the binary (FLAC, Vorbis, Opus and mpg123 report their versions in the binary; |
| LAME's is pinned by the libsndfile-binaries build script; libogg's COPYING is the same in 1.3.4, 1.3.5 and 1.3.6). |
|
|
| `anuvani_diar/assets/`: the PLDA and x-vector arrays (`*.bin`), `MANIFEST.json` (their sha256), `nn_golden.json`, |
| `long_golden.json`; about 270 KB. |
|
|
| **Staged, not committed.** Both folders are written by `tools/stage_wheel_files.py` at build time and are git-ignored. |
| The libraries are 41.7 MB of third-party binaries (and must not be in git); the assets already live in `assets/`, where |
| the Rust tests and the gateway read them, and a second committed copy could drift. The script copies the libraries from |
| the reference environment's site-packages and the codec licence texts from `tools/wheel_licences/`, checks every file |
| against `tools/wheel_files.json` (the committed sha256 manifest) and the arrays against `assets/MANIFEST.json`, and |
| leaves exactly the listed files. `make build`, `make test-rust` and `tools/build_wheel.sh` run it. |
|
|
| ## How the engine finds them |
|
|
| `anuvani_diar/__init__.py` sets three variables to the package's own copies before `_core` loads, unless they are set |
| (non-empty) already: |
|
|
| | Variable | Default (set by the package) | Used by | |
| |---|---|---| |
| | `ANUVANI_DIAR_OPENBLAS_DIR` | `<package>/_libs` | `crates/diar-exact/src/blas.rs` | |
| | `ANUVANI_DIAR_SNDFILE` | `<package>/_libs/libsndfile_x86_64.so` | `crates/diar-audio/src/sndfile.rs` | |
| | `ANUVANI_DIAR_ASSETS` | `<package>/assets` | `crates/diar-core/src/assets.rs` | |
| | `ANUVANI_DIAR_SNDFILE_UNCHECKED` | unset | `1`: load the libsndfile named by `ANUVANI_DIAR_SNDFILE` without the sha256 and version checks | |
|
|
| No path is compiled into `_core`. Without the variables (`cargo test`, the gateway under `target/`, a source checkout) |
| the loaders fall back, for development only, to the source tree found at run time (from cargo's `CARGO_MANIFEST_DIR`, |
| the running executable, or the extension's own file): its staged `python/anuvani_diar/_libs` and its `assets/`. |
|
|
| **OpenBLAS.** The three files must have their sha256 before anything is loaded. OpenBLAS reads its settings from the |
| environment while it loads, so the loader sets `OPENBLAS_NUM_THREADS=1` and, if the CPU reports AVX2 and FMA, |
| `OPENBLAS_CORETYPE=Zen` just before the first `dlmopen`, restores both afterwards, and then requires 1 thread and core |
| "Zen". On a CPU without AVX2 or FMA the core is not forced and loading fails with "exact mode unavailable" (the Zen |
| kernels need both; other kernels would not reproduce the release bit for bit). The private namespace keeps its own |
| copy of the environment, so a later `setenv` elsewhere cannot leave it reading freed memory. If glibc has no static TLS |
| space left for the namespace ("cannot allocate memory in static TLS block"), the error says so and names the remedy: |
| start the process with `GLIBC_TUNABLES=glibc.rtld.optional_static_tls=4096`, or load the engine's OpenBLAS first |
| (`anuvani_diar._core.openblas_info()`). `_core.openblas_info()` reports the file, threads, core and whether the core |
| was forced; `_core.sndfile_info()` the libsndfile file, version and whether it passed the checks. |
|
|
| **libsndfile.** Checked by sha256 and version (`libsndfile-1.2.2`). The LGPL lets users replace it: |
| `ANUVANI_DIAR_SNDFILE=/path/to/libsndfile.so.1 ANUVANI_DIAR_SNDFILE_UNCHECKED=1` loads another build, any version, |
| and the engine says on stderr that decoding is no longer claimed to be identical to the release's. |
|
|
| ## Building |
|
|
| ```bash |
| # about 20 s once the cache is warm; the first build also fetches the toolchain and crates |
| tools/build_wheel.sh |
| ``` |
|
|
| Pinned: the image `quay.io/pypa/manylinux_2_28_x86_64@sha256:39df0042d5cc900b085aa25a0659368b42a0006c54c474299b785b44c1b4ff82` |
| (AlmaLinux 8, glibc 2.28; pulled 9 October 2026, created 3 October 2026), rustup-init 1.29.1 (by sha256), rustc 1.94.1 |
| (minimal profile), maturin 1.15.0, the crates by `Cargo.lock` (`--locked`), and file times by `SOURCE_DATE_EPOCH` |
| (the commit's time). The build context is the tracked build files plus the staged ones, copied to |
| `artifacts/wheel_build/`; the target directory starts empty. maturin runs with `--strip`, every source path remapped |
| (`/build/anuvani_diar_rs` to `/anuvani_diar_rs`, the crate registry to `/cargo`), `--compatibility manylinux_2_28` and |
| `--auditwheel check` (it verifies the tag and changes nothing). `dlmopen` and `dlinfo` are declared with |
| `#[link(name = "dl")]` in `blas.rs`, so on glibc 2.28 they come from `libdl.so.2`; on newer glibc the link finds only |
| an empty `libdl.a` and nothing changes. Strip and remapping are set in the script, not in `pyproject.toml`, so that |
| `maturin develop` keeps its symbols. `pyproject.toml` names the staged folders in `include` (maturin leaves git-ignored |
| files out otherwise). Caches (rustup, cargo, maturin) stay in `artifacts/wheel_build/cache`. |
|
|
| **Reproducible:** two builds from empty target directories at the same commit gave the same wheel, byte for byte |
| (`acd8a022…` twice, 9 October 2026); a new commit changes only the file times. |
|
|
| **Checks made by the build** (all passed for the wheel above): |
|
|
| - the bundled files in the wheel have the sha256 of `tools/wheel_files.json` and `assets/MANIFEST.json`; |
| - `strings` on `_core.abi3.so`: no path of the build machine or of the build. What remains is the Rust toolchain's own |
| (`/rustc/<hash>/library/...`, `/rust/deps/...`, `/checkout/src`), the remapped registry (`/cargo/registry/src/...`), |
| the project's relative `crates/...` and system files the standard library reads (`/proc/self/...`, `/dev/urandom`); |
| - `_core` needs glibc symbols up to 2.28 and only `libdl.so.2`, `libgcc_s.so.1`, `libpthread.so.0`, `libm.so.6`, |
| `libc.so.6`, `ld-linux-x86-64.so.2`; |
| - `auditwheel show` (never `repair`, which would rewrite the bundled libraries and break their sha256): the symbol |
| versions allow `manylinux_2_28_x86_64`, but its overall verdict is `linux_x86_64`, for one reason: OpenBLAS needs |
| `libgfortran-040039e1.so.5.0.0` and has no RPATH, so auditwheel cannot see that the file is in the wheel (the engine |
| loads it first, by path, into the same namespace). Every other library the bundled files need is on the manylinux |
| list (`libz.so.1`, `libmvec.so.1`, `libgcc_s.so.1`, glibc's). Making auditwheel agree would mean editing OpenBLAS. |
| The SBOM maturin adds (`.dist-info/sboms/diar-py.cyclonedx.json`) names the build-container path |
| `/build/anuvani_diar_rs`, not a host path. |
|
|
| ## What was tested |
|
|
| `tools/wheel_check.py` (CPU only, `CUDA_VISIBLE_DEVICES` empty), on this wheel, 9 October 2026: **PASS in 32.6 s** |
| ((a) and (b) side by side; 42.9 s on the first run, which also fetched Python 3.12 and torch). |
|
|
| (a) In `.venv` with the canonical environment, the existing tests that cover the bundled libraries and assets: |
| decoding against soundfile over every format and 2,750 damaged files, the release's `load_audio`, numpy's memory-error |
| text, the VBx row-parallel products against numpy's own (shapes up to 5,000 rows), the engine with every BLAS thread |
| variable at 64, and 40 cached records' RTTMs against the canonical oracle through `diarize_record` and through the |
| engine: **121 passed** (24 of the 20,774-row product shapes left to `make test-fast`). From a git worktree the main |
| checkout's untracked data are bound read-only over the worktree's in a private mount namespace. |
|
|
| (b) The wheel in a fresh venv: Python 3.12.11, torch 2.11.0+cpu, numpy 2.5.3, no variable of `env/canonical.sh` and |
| no `ANUVANI_DIAR_*` variable set; torch and numpy imported before `anuvani_diar`. |
|
|
| | Check | Result | |
| |---|---| |
| | private OpenBLAS | from the package's `_libs`; 1 thread, core Zen (forced), 0 threads started by the load (0.075 s) | |
| | libsndfile | from the package's `_libs`; `libsndfile-1.2.2`, checks passed | |
| | bengali_009.flac, telugu_020.flac (golden inputs, 3,200,512 samples each at 16 kHz) | sha256 of the decoded float32 samples identical to soundfile 0.14.0's in the reference environment | |
| | long_1h record, 2,241 chunks | RTTM sha256 `0f74418f…` = `assets/long_golden.json` (1.07 s) | |
| | long_3h record, 6,741 chunks | `982bbb38…` = golden (6.24 s) | |
| | long_6h record, 13,491 chunks | `651635aa…` = golden (19.30 s) | |
|
|
| Also checked while making the change (before and after it, on the build machine): |
|
|
| - the whole `make test-fast` from this worktree (data bound read-only as above), at commit 5ed8a2fd: 462 passed in |
| 7 min 23 s, the Rust tests included; |
| - the same test files without the 106 s channel-mean property test and the slow sets: 247 passed before the change |
| and 247 after; |
| - `cargo test --workspace`: all pass, with new tests for the hash refusal (OpenBLAS and libsndfile), the static-TLS |
| message, the core pin (forcing Haswell shows up as core "Haswell", so the pin reaches OpenBLAS), the namespace's own |
| environment copy, and the source-tree fallback (also from a test binary run with an empty environment); |
| - loaded without any OpenBLAS variable set, the private OpenBLAS starts no thread and leaves both variables unset |
| again; the same binary loaded without the pin (numpy importing it) starts 63 extra threads on this 64-CPU host; |
| - static TLS used up on purpose (`glibc.rtld.nns=1`, `optional_static_tls=0`, two libgomp copies loaded first): the |
| readable error appears; with `optional_static_tls=4096` the same process loads; |
| - two gateway tests with the stand-in Triton (every RTTM against the release, the resample limit). |
|
|
| ## Not covered here |
|
|
| - The networks: since 9 October 2026 the wheel declares torch, safetensors and huggingface_hub as well (the networks, |
| their weights, the download). |
| - Licences: the code is MIT OR Apache-2.0 (decided 10 October 2026). The wheel carries `LICENSE-MIT`, |
| `LICENSE-APACHE`, `THIRD_PARTY_NOTICES.md` and the licence files of every Rust crate compiled into `_core` |
| (`LICENSES/crates/`, from `tools/third_party_notices.py`) in its `.dist-info/licenses/`, beside the bundled |
| libraries' texts in `_libs/LICENSES/`. The source of the LGPL libraries is in the model repository's |
| `third_party_sources/`. |
| - Other CPUs and older glibc: tested on one AVX2 CPU with glibc 2.35 only. |
|
|
