Hugging Face's logo

CIIL
/
Anuvani-Diarization

Voice Activity Detection
Safetensors
PyTorch
Rust
speaker-diarization
audio
indic
diarizen
wavlm
speech
speaker
speaker-segmentation
overlapped-speech-detection
conformer
wespeaker
vbx
Eval Results (legacy)
Model card Files
xet
 Community
Anuvani-Diarization / src /triton_inference /docs /reference /06-tensorrt-build.md
StephennFernandes's picture
StephennFernandes
Anuvani-Diarization 1.0.0
63dd946
preview code
|
Raw History Blame Contribute Delete
14.7 kB

TensorRT for the diarization networks

What TensorRT offers and demands for the two networks (pruned WavLM-Large + Conformer segmentation; fbank + ResNet34 voiceprints): typing, precision, shape profiles, attention, normalisation, the ONNX operators the front ends need, build settings, portability and accuracy tools. Facts carry the TensorRT release they come from: 10.5 (Triton 24.10), 10.11 (25.06), 10.16 (26.05) or 11.x (documentation set 11.4.0, dated 5 October 2026). Checked 9 October 2026.

Which TensorRT each image carries

  • Triton 24.10 ships TensorRT 10.5.0.18, 25.06 ships 10.11.0.33 and 26.05 ships 10.16.1.11 [25], so none of these images carries TensorRT 11 (inference); 15-triton-image-versions.md covers the images.

Strong typing: TensorRT 10 against TensorRT 11

  • Strongly typed: "TensorRT's optimizer will statically infer intermediate tensor types based on the network input types and the operator specifications". Weakly typed: the optimiser "can substitute different precisions for tensors ... if it increases performance", defaulting to FP32 for floating-point operations [26].
  • In TensorRT 10 strong typing is opt-in: NetworkDefinitionCreationFlag.STRONGLY_TYPED, or trtexec --stronglyTyped, with which "builder precision flags such as --int8 or --best ... is not allowed" [7].
  • TensorRT 10.12 deprecated the weak-typing APIs: BuilderFlag kINT8, kFP8, kFP16, kBF16, the two precision-constraint flags, ILayer::setPrecision and ILayer::setOutputType [2]. They still work in 10.5 and 10.11.
  • TensorRT 11.0: "Strongly typed networks are now the default" and "Weak typing is no longer supported"; the per-precision BuilderFlags (kFP16, kBF16, kFP8, kINT8, kINT4, kFP4, kOBEY/kPREFER_PRECISION_CONSTRAINTS), setType, setDynamicRange, setPrecision and setOutputType were removed, as were implicit quantisation and IInt8Calibrator [1].
  • trtexec in 11.x removed --fp16, --bf16, --int8, --fp8, --int4, --best, --precisionConstraints, --layerPrecisions, --layerOutputTypes and --calib; --stronglyTyped is a no-op; CUDA graphs, spin-wait and the separate profile run are on by default [4, 6]. The replacement for --fp16 is ModelOpt AutoCast on the ONNX file (python -m modelopt.onnx.autocast --onnx_path model.onnx), or Cast nodes / addCast placed by hand [3, 4].
  • kTF32 survives: it "is permitted as it controls TF32 Tensor Core usage for FP32 types" [5].

Precision on the target GPUs

  • Support matrix (same rows in every 10.x release and in 11.4): compute capability 8.6 supports FP32, TF32, FP16, BF16 and INT8, with FP8 and FP4 not applicable; 7.5 (T4) supports FP32, FP16 and INT8, with TF32 and BF16 not applicable [8].
  • TF32 is on by default: "TF32, enabled by default in TensorRT, uses an 8-bit exponent and a 10-bit mantissa" [9]; BuilderFlag.TF32 "Allow (but not require) computations on tensors of type DataType.FLOAT to use TF32", enabled by default [10]. Turn it off with trtexec --noTF32 ("default is to enable tf32, in addition to fp32") [6] or config.clear_flag(trt.BuilderFlag.TF32), which is what Polygraphy does [22].

Optimisation profiles for the window batch

  • A profile gives MIN, OPT and MAX dimensions per input (trtexec --minShapes= --optShapes= --maxShapes= [6, 7]); OPT is the shape the auto-tuner optimises for, and "Some operations have tactics that only work for MIN=OPT=MAX, so when these values differ, the tactic is disabled"; memory use can change sharply with the three values [12].
  • Each execution context must use its own optimisation profile; with dynamic inputs the profile index must not be in use by another live context; select it with setOptimizationProfileAsync(index, stream) [12].

Attention and normalisation

  • Fusion target: softmax(Q·Kᵀ ⊗ mask)·V, built with IAttention (recommended; Q or K pre-multiplied by the attention scale, or both by its square root) or with MatMul, Softmax and MatMul layers [14]. IAttention arrived in 10.14.1 [16].
  • Fused attention on SM75-SM90: FP16 with 16 ≤ H ≤ 256 and H % 8 == 0, no sequence-length limit; BF16 on SM80+ with the same head rule; INT8 only for H in {16, 32, 64} and S ≤ 512; FP8 on SM89+ only [15]. The tables have no FP32 or TF32 column [14, 15]. An IAttention with no fused kernel fails the build unless setDecomposable is called [15]. Check with trtexec --loadEngine=model.engine --profilingVerbosity=detailed --dumpLayerInfo --skipInference and look for the mha op [14].
  • ONNX LayerNormalization is supported for FP32, FP16 and BF16, first output only [17]. PyTorch 2.1.1 emits it from its opset-17 exporter (aten::layer_norm) [27]; the 10.5 parser accepts opsets 9 to 20 [17].
  • INormalizationLayer: "The compute precision defaults to DataType::kFLOAT32", a setting that exists "to avoid overflow errors ... in mixed precision mode"; a strongly typed network rejects setComputePrecision "since the compute precision is typically controlled by casting the input tensors"; deprecated in 10.16 [13].

Convolutions and the front ends

  • TensorRT 10.0 disabled and deprecated the cuDNN and cuBLAS tactic sources [11]; in 10.16.1 cuBLAS, cuBLASLt and cuDNN are off by default, edge-mask and JIT convolutions on [10]. TensorRT 11.0 dropped plugins that need cuDNN [1].
  • 1-D convolutions (the 7-layer waveform front end): the 10.5 parser unsqueezes a rank-3 Conv input to 2-D ("Expand spatial dims from 1D to 2D") and squeezes back [18].
  • Fbank operators: ONNX DFT is unsupported in the 10.5 and 10.11 matrices and supported in 11.3 through a cuFFT-backed plugin (axis -2; FP16/BF16 need power-of-two lengths) [17]. STFT is FP32-only in 10.x (initialiser frame_step and window, real input), FP32/FP16/BF16 in 11.3 [17], and the 10.5 parser builds it "via decomposion into 1D convs" [18]. MelWeightMatrix is unsupported throughout [17].
  • kaldi.fbank runs torch.fft.rfft, abs, pow(2), a torch.mm with the mel banks and max(eps).log() [28]; pyannote's WeSpeaker wrapper maps it over windows with torch.vmap [29].

Build settings and portability

  • The timing cache is "specific to the targeted device, CUDA, TensorRT versions, and BuilderConfig parameters that can change the layer implementation, such as BuilderFlag::kTF32 or BuilderFlag::kREFIT" [19]; save and load it with --timingCacheFile=<file> [6]. Optimisation level: default 3, valid 0 to 5; lower builds much faster but the engine can be slower, higher builds longer and can find better tactics [6, 19].
  • "By default, TensorRT engines are compatible only with the version of TensorRT used to build them" and "only compatible with the type of device where they were built" [20]. Version-compatible engines (--versionCompatible) embed a lean runtime, run TensorRT 10 engines on an 11 runtime, and "can be slower" [20, 6]. Hardware compatibility kAMPERE_PLUS (10.5 trtexec: none|ampere+) drops architecture-specific tactics; kSAME_COMPUTE_CAPABILITY (11.x) is generally faster than kAMPERE_PLUS [20, 6].

Checking accuracy with Polygraphy

  • polygraphy run model.onnx --trt --onnxrt --input-shapes X:[...] compares TensorRT with ONNX-Runtime; --atol, --rtol and --check-error-stat elemwise|mean|median|max set the test [21]. Per-layer: --trt-outputs mark all --onnx-outputs mark all (10.5) or '*' (11.3), which "can sometimes perturb the generated engine" [21].
  • Polygraphy leaves TF32 off unless --tf32 is given ("TF32 is disabled by default") [22], the opposite of trtexec [6].
  • polygraphy debug precision ("[EXPERIMENTAL] Iteratively mark layers to run in a higher precision") sets layer.precision [23], an API TensorRT 11 removed [1], so it needs TensorRT 10 (inference).

Unverified

  • Whether torch.onnx 2.1.1 exports torch.fft.rfft or torch.vmap (no rfft symbolic in opsets 9, 13, 17; rest unread).
  • The fused-attention table as printed for TensorRT 10.5 itself ([14] is the 10.x set at 10.16.1).
  • Whether WavLM's gated relative position bias, an additive per-head T x T term, still matches the fusion pattern.

What this means for the diarization model

  • Engines must be built with the TensorRT of the serving image, on the GPU model that will run them: 10.5.0.18 for 24.10, rebuilt for 10.16.1.11 if serving moves to 26.05 (inference from [20, 25]). A T4 has no TF32 or BF16 path [8].
  • An "FP32" engine is TF32 unless built with --noTF32, and even then it is a different kernel set from PyTorch 2.1.1, so TensorRT belongs to the faster lane, never the exact lane (inference from [6, 9]).
  • WavLM's head size is 64 (1024 / 16 heads upstream [24]), inside the FP16 fused range and, on SM80+, the BF16 range; 799 frames are above the INT8 limit of 512 [15]. Per-layer head counts of 0 to 10 give a different attention shape per layer, and the four attention-free layers have nothing to fuse (inference).
  • Keep the fbank in PyTorch FP32 and hand the ResNet34 engine (B, frames, 80) features: DFT is unsupported in the 10.x parser and the vmap export is unverified (inference from [17, 28, 29]).
  • Profile the window batch with OPT = 8 (one block) and MAX = the largest packed request; a static profile of exactly 8 keeps MIN=OPT=MAX tactics; two CUDA streams need two TensorRT execution contexts and two profiles (inference from [12]). Localise divergence with Polygraphy per-layer comparisons against ONNX-Runtime FP32; DER decides (inference).

Sources

  1. TensorRT 11.0.0 Release Notes - https://docs.nvidia.com/deeplearning/tensorrt/latest/getting-started/release-notes-11/11.0.0.html - TensorRT 11.0.0.
  2. TensorRT 10.12.0 Release Notes - https://docs.nvidia.com/deeplearning/tensorrt/10.x.x/getting-started/release-notes-10/10.12.0.html - TensorRT 10.12.0.
  3. Migrating from TensorRT 10.x to 11.x - https://docs.nvidia.com/deeplearning/tensorrt/latest/api/migration/tensorrt-10x-to-11x.html - TensorRT 11.x (docs 11.4.0).
  4. trtexec migration 10.x to 11.x - https://docs.nvidia.com/deeplearning/tensorrt/latest/api/migration/tensorrt-10x-to-11x-trtexec.html - TensorRT 11.x.
  5. Precision Control - https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/precision-control.html - TensorRT 11.x (docs 11.4.0).
  6. trtexec help text, samples/common/sampleOptions.cpp - https://github.com/NVIDIA/TensorRT/blob/release/10.5/samples/common/sampleOptions.cpp and https://github.com/NVIDIA/TensorRT/blob/release/11.3/samples/common/sampleOptions.cpp - TensorRT OSS release/10.5 and release/11.3.
  7. trtexec README - https://github.com/NVIDIA/TensorRT/blob/release/10.5/samples/trtexec/README.md - release/10.5.
  8. Support Matrix, GPU architecture and precision support - https://docs.nvidia.com/deeplearning/tensorrt/latest/getting-started/support-matrix.html and https://docs.nvidia.com/deeplearning/tensorrt/10.x.x/getting-started/support-matrix.html - TensorRT 11.4.0 and 10.0.0 EA to 10.16.1.
  9. Accuracy Considerations - https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/accuracy-considerations.html - TensorRT 11.4.0.
  10. Python API, BuilderConfig - https://docs.nvidia.com/deeplearning/tensorrt/10.x.x/_static/python-api/infer/Core/BuilderConfig.html - TensorRT 10.16.1.
  11. TensorRT 10.0.0 EA Release Notes - https://docs.nvidia.com/deeplearning/tensorrt/10.x.x/getting-started/release-notes-10/10.0.0-ea.html - TensorRT 10.0.0 EA.
  12. Dynamic Shapes: Core Concepts - https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/dynamic-shapes-basics.html - TensorRT 11.4.0.
  13. INormalizationLayer (C++ API; Python API Layers page) - https://docs.nvidia.com/deeplearning/tensorrt/10.x.x/_static/c-api/classnvinfer1_1_1_i_normalization_layer.html and https://docs.nvidia.com/deeplearning/tensorrt/10.x.x/_static/python-api/infer/Graph/Layers.html - TensorRT 10.16.1.
  14. Working with Transformers (multi-head attention fusion) - https://docs.nvidia.com/deeplearning/tensorrt/10.x.x/inference-library/work-with-transformers.html - TensorRT 10.x docs set (10.16.1).
  15. Fused Attention - https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/transformers-fused-attention.html - TensorRT 11.4.0.
  16. TensorRT 10.14.1 Release Notes - https://docs.nvidia.com/deeplearning/tensorrt/10.x.x/getting-started/release-notes-10/10.14.1.html - TensorRT 10.14.1.
  17. onnx-tensorrt Supported ONNX Operators - https://github.com/onnx/onnx-tensorrt/blob/10.5-GA/docs/operators.md (header reads TensorRT 10.4, opsets 9-20), https://github.com/onnx/onnx-tensorrt/blob/10.11-GA/docs/operators.md (10.11, opsets 9-23), https://github.com/onnx/onnx-tensorrt/blob/main/docs/operators.md (11.3, opsets 9-24).
  18. onnx-tensorrt Conv and STFT importers - https://github.com/onnx/onnx-tensorrt/blob/10.5-GA/onnxOpImporters.cpp - branch 10.5-GA.
  19. Optimizing Builder Performance - https://docs.nvidia.com/deeplearning/tensorrt/10.x.x/performance/builder-performance.html - TensorRT 10.x docs set (10.16.1).
  20. Engine Compatibility - https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/engine-compatibility.html - TensorRT 11.4.0.
  21. Polygraphy example, Comparing Frameworks - https://github.com/NVIDIA/TensorRT/blob/release/10.5/tools/Polygraphy/examples/cli/run/01_comparing_frameworks/README.md and the release/11.3 copy - release/10.5 and release/11.3.
  22. Polygraphy CreateConfig source - https://github.com/NVIDIA/TensorRT/blob/release/11.3/tools/Polygraphy/polygraphy/backend/trt/config.py - release/11.3.
  23. Polygraphy debug precision source - https://github.com/NVIDIA/TensorRT/blob/release/11.3/tools/Polygraphy/polygraphy/tools/debug/subtool/precision.py - release/11.3.
  24. microsoft/wavlm-large config.json - https://huggingface.co/microsoft/wavlm-large/blob/main/config.json - main branch.
  25. Triton release notes 24.10, 25.06, 26.05 - https://docs.nvidia.com/deeplearning/triton-inference-server/release-notes/rel-24-10.html, .../rel-25-06.html, .../rel-26-05.html - those releases.
  26. TensorRT capabilities, Strong Typing vs Weak Typing - https://docs.nvidia.com/deeplearning/tensorrt/10.x.x/architecture/capabilities.html - TensorRT 10.x docs set (10.16.1).
  27. PyTorch ONNX exporter, symbolic_opset17.py - https://github.com/pytorch/pytorch/blob/v2.1.1/torch/onnx/symbolic_opset17.py - PyTorch v2.1.1.
  28. torchaudio kaldi.py - https://github.com/pytorch/audio/blob/v2.1.1/torchaudio/compliance/kaldi.py - torchaudio v2.1.1.
  29. pyannote WeSpeaker wrapper - https://github.com/pyannote/pyannote-audio/blob/3.1.1/pyannote/audio/models/embedding/wespeaker/__init__.py - pyannote-audio 3.1.1.