# TensorRT for the diarization networks

What TensorRT offers and demands for the two networks (pruned WavLM-Large + Conformer segmentation; fbank + ResNet34
voiceprints): typing, precision, shape profiles, attention, normalisation, the ONNX operators the front ends need,
build settings, portability and accuracy tools. Facts carry the TensorRT release they come from: 10.5 (Triton 24.10),
10.11 (25.06), 10.16 (26.05) or 11.x (documentation set 11.4.0, dated 5 October 2026). Checked 9 October 2026.

## Which TensorRT each image carries

- Triton 24.10 ships TensorRT 10.5.0.18, 25.06 ships 10.11.0.33 and 26.05 ships 10.16.1.11 [25], so none of these images
  carries TensorRT 11 (inference); 15-triton-image-versions.md covers the images.

## Strong typing: TensorRT 10 against TensorRT 11

- Strongly typed: "TensorRT's optimizer will statically infer intermediate tensor types based on the network input
  types and the operator specifications". Weakly typed: the optimiser "can substitute different precisions for tensors
  ... if it increases performance", defaulting to FP32 for floating-point operations [26].
- In TensorRT 10 strong typing is opt-in: `NetworkDefinitionCreationFlag.STRONGLY_TYPED`, or `trtexec --stronglyTyped`,
  with which "builder precision flags such as `--int8` or `--best` ... is not allowed" [7].
- TensorRT 10.12 deprecated the weak-typing APIs: BuilderFlag kINT8, kFP8, kFP16, kBF16, the two precision-constraint
  flags, `ILayer::setPrecision` and `ILayer::setOutputType` [2]. They still work in 10.5 and 10.11.
- TensorRT 11.0: "Strongly typed networks are now the default" and "Weak typing is no longer supported"; the
  per-precision BuilderFlags (kFP16, kBF16, kFP8, kINT8, kINT4, kFP4, kOBEY/kPREFER_PRECISION_CONSTRAINTS),
  `setType`, `setDynamicRange`, `setPrecision` and `setOutputType` were removed, as were implicit quantisation and
  `IInt8Calibrator` [1].
- trtexec in 11.x removed `--fp16`, `--bf16`, `--int8`, `--fp8`, `--int4`, `--best`, `--precisionConstraints`,
  `--layerPrecisions`, `--layerOutputTypes` and `--calib`; `--stronglyTyped` is a no-op; CUDA graphs, spin-wait and the
  separate profile run are on by default [4, 6]. The replacement for `--fp16` is ModelOpt AutoCast on the ONNX file
  (`python -m modelopt.onnx.autocast --onnx_path model.onnx`), or Cast nodes / `addCast` placed by hand [3, 4].
- `kTF32` survives: it "is permitted as it controls TF32 Tensor Core usage for FP32 types" [5].

## Precision on the target GPUs

- Support matrix (same rows in every 10.x release and in 11.4): compute capability 8.6 supports FP32, TF32, FP16, BF16
  and INT8, with FP8 and FP4 not applicable; 7.5 (T4) supports FP32, FP16 and INT8, with TF32 and BF16 not
  applicable [8].
- TF32 is on by default: "TF32, enabled by default in TensorRT, uses an 8-bit exponent and a 10-bit mantissa" [9];
  `BuilderFlag.TF32` "Allow (but not require) computations on tensors of type DataType.FLOAT to use TF32", enabled by
  default [10]. Turn it off with `trtexec --noTF32` ("default is to enable tf32, in addition to fp32") [6] or
  `config.clear_flag(trt.BuilderFlag.TF32)`, which is what Polygraphy does [22].

## Optimisation profiles for the window batch

- A profile gives MIN, OPT and MAX dimensions per input (`trtexec --minShapes= --optShapes= --maxShapes=` [6, 7]);
  OPT is the shape the auto-tuner optimises for, and "Some operations have tactics that only work for MIN=OPT=MAX, so
  when these values differ, the tactic is disabled"; memory use can change sharply with the three values [12].
- Each execution context must use its own optimisation profile; with dynamic inputs the profile index must not be in
  use by another live context; select it with `setOptimizationProfileAsync(index, stream)` [12].

## Attention and normalisation

- Fusion target: softmax(Q·Kᵀ ⊗ mask)·V, built with `IAttention` (recommended; Q or K pre-multiplied by the attention
  scale, or both by its square root) or with MatMul, Softmax and MatMul layers [14]. `IAttention` arrived in 10.14.1 [16].
- Fused attention on SM75-SM90: FP16 with 16 ≤ H ≤ 256 and H % 8 == 0, no sequence-length limit;
  BF16 on SM80+ with the same head rule; INT8 only for H in {16, 32, 64} and S ≤ 512; FP8 on SM89+ only [15]. The
  tables have no FP32 or TF32 column [14, 15]. An `IAttention` with no fused kernel fails the build unless
  `setDecomposable` is called [15]. Check with `trtexec --loadEngine=model.engine --profilingVerbosity=detailed
  --dumpLayerInfo --skipInference` and look for the `mha` op [14].
- ONNX `LayerNormalization` is supported for FP32, FP16 and BF16, first output only [17]. PyTorch 2.1.1 emits it from
  its opset-17 exporter (`aten::layer_norm`) [27]; the 10.5 parser accepts opsets 9 to 20 [17].
- `INormalizationLayer`: "The compute precision defaults to DataType::kFLOAT32", a setting that exists "to avoid
  overflow errors ... in mixed precision mode"; a strongly typed network rejects `setComputePrecision` "since the
  compute precision is typically controlled by casting the input tensors"; deprecated in 10.16 [13].

## Convolutions and the front ends

- TensorRT 10.0 disabled and deprecated the cuDNN and cuBLAS tactic sources [11]; in 10.16.1 cuBLAS, cuBLASLt and
  cuDNN are off by default, edge-mask and JIT convolutions on [10]. TensorRT 11.0 dropped plugins that need cuDNN [1].
- 1-D convolutions (the 7-layer waveform front end): the 10.5 parser unsqueezes a rank-3 Conv input to 2-D ("Expand
  spatial dims from 1D to 2D") and squeezes back [18].
- Fbank operators: ONNX `DFT` is unsupported in the 10.5 and 10.11 matrices and supported in 11.3 through a
  cuFFT-backed plugin (axis -2; FP16/BF16 need power-of-two lengths) [17]. `STFT` is FP32-only in 10.x (initialiser
  frame_step and window, real input), FP32/FP16/BF16 in 11.3 [17], and the 10.5 parser builds it "via decomposion into
  1D convs" [18]. `MelWeightMatrix` is unsupported throughout [17].
- `kaldi.fbank` runs `torch.fft.rfft`, `abs`, `pow(2)`, a `torch.mm` with the mel banks and `max(eps).log()` [28];
  pyannote's WeSpeaker wrapper maps it over windows with `torch.vmap` [29].

## Build settings and portability

- The timing cache is "specific to the targeted device, CUDA, TensorRT versions, and BuilderConfig parameters that
  can change the layer implementation, such as BuilderFlag::kTF32 or BuilderFlag::kREFIT" [19]; save and load it with
  `--timingCacheFile=<file>` [6]. Optimisation level: default 3, valid 0 to 5; lower builds much faster but the engine
  can be slower, higher builds longer and can find better tactics [6, 19].
- "By default, TensorRT engines are compatible only with the version of TensorRT used to build them" and "only
  compatible with the type of device where they were built" [20]. Version-compatible engines (`--versionCompatible`)
  embed a lean runtime, run TensorRT 10 engines on an 11 runtime, and "can be slower" [20, 6]. Hardware compatibility
  `kAMPERE_PLUS` (10.5 trtexec: `none|ampere+`) drops architecture-specific tactics; `kSAME_COMPUTE_CAPABILITY` (11.x)
  is generally faster than `kAMPERE_PLUS` [20, 6].

## Checking accuracy with Polygraphy

- `polygraphy run model.onnx --trt --onnxrt --input-shapes X:[...]` compares TensorRT with ONNX-Runtime; `--atol`,
  `--rtol` and `--check-error-stat elemwise|mean|median|max` set the test [21]. Per-layer: `--trt-outputs mark all
  --onnx-outputs mark all` (10.5) or `'*'` (11.3), which "can sometimes perturb the generated engine" [21].
- Polygraphy leaves TF32 off unless `--tf32` is given ("TF32 is disabled by default") [22], the opposite of trtexec [6].
- `polygraphy debug precision` ("[EXPERIMENTAL] Iteratively mark layers to run in a higher precision") sets
  `layer.precision` [23], an API TensorRT 11 removed [1], so it needs TensorRT 10 (inference).

## Unverified

- Whether `torch.onnx` 2.1.1 exports `torch.fft.rfft` or `torch.vmap` (no rfft symbolic in opsets 9, 13, 17; rest unread).
- The fused-attention table as printed for TensorRT 10.5 itself ([14] is the 10.x set at 10.16.1).
- Whether WavLM's gated relative position bias, an additive per-head T x T term, still matches the fusion pattern.

## What this means for the diarization model

- Engines must be built with the TensorRT of the serving image, on the GPU model that will run them: 10.5.0.18 for
  24.10, rebuilt for 10.16.1.11 if serving moves to 26.05 (inference from [20, 25]). A T4 has no TF32 or BF16 path
  [8].
- An "FP32" engine is TF32 unless built with `--noTF32`, and even then it is a different kernel set from PyTorch 2.1.1,
  so TensorRT belongs to the faster lane, never the exact lane (inference from [6, 9]).
- WavLM's head size is 64 (1024 / 16 heads upstream [24]), inside the FP16 fused range and, on SM80+, the BF16 range;
  799 frames are above the INT8 limit of 512 [15]. Per-layer head counts of 0 to 10 give a different attention shape
  per layer, and the four attention-free layers have nothing to fuse (inference).
- Keep the fbank in PyTorch FP32 and hand the ResNet34 engine (B, frames, 80) features: DFT is unsupported in the
  10.x parser and the vmap export is unverified (inference from [17, 28, 29]).
- Profile the window batch with OPT = 8 (one block) and MAX = the largest packed request; a static profile of exactly
  8 keeps MIN=OPT=MAX tactics; two CUDA streams need two TensorRT execution contexts and two profiles (inference from
  [12]). Localise divergence with Polygraphy per-layer comparisons against ONNX-Runtime FP32; DER decides (inference).

## Sources

1. TensorRT 11.0.0 Release Notes - https://docs.nvidia.com/deeplearning/tensorrt/latest/getting-started/release-notes-11/11.0.0.html - TensorRT 11.0.0.
2. TensorRT 10.12.0 Release Notes - https://docs.nvidia.com/deeplearning/tensorrt/10.x.x/getting-started/release-notes-10/10.12.0.html - TensorRT 10.12.0.
3. Migrating from TensorRT 10.x to 11.x - https://docs.nvidia.com/deeplearning/tensorrt/latest/api/migration/tensorrt-10x-to-11x.html - TensorRT 11.x (docs 11.4.0).
4. trtexec migration 10.x to 11.x - https://docs.nvidia.com/deeplearning/tensorrt/latest/api/migration/tensorrt-10x-to-11x-trtexec.html - TensorRT 11.x.
5. Precision Control - https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/precision-control.html - TensorRT 11.x (docs 11.4.0).
6. trtexec help text, samples/common/sampleOptions.cpp - https://github.com/NVIDIA/TensorRT/blob/release/10.5/samples/common/sampleOptions.cpp and https://github.com/NVIDIA/TensorRT/blob/release/11.3/samples/common/sampleOptions.cpp - TensorRT OSS release/10.5 and release/11.3.
7. trtexec README - https://github.com/NVIDIA/TensorRT/blob/release/10.5/samples/trtexec/README.md - release/10.5.
8. Support Matrix, GPU architecture and precision support - https://docs.nvidia.com/deeplearning/tensorrt/latest/getting-started/support-matrix.html and https://docs.nvidia.com/deeplearning/tensorrt/10.x.x/getting-started/support-matrix.html - TensorRT 11.4.0 and 10.0.0 EA to 10.16.1.
9. Accuracy Considerations - https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/accuracy-considerations.html - TensorRT 11.4.0.
10. Python API, BuilderConfig - https://docs.nvidia.com/deeplearning/tensorrt/10.x.x/_static/python-api/infer/Core/BuilderConfig.html - TensorRT 10.16.1.
11. TensorRT 10.0.0 EA Release Notes - https://docs.nvidia.com/deeplearning/tensorrt/10.x.x/getting-started/release-notes-10/10.0.0-ea.html - TensorRT 10.0.0 EA.
12. Dynamic Shapes: Core Concepts - https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/dynamic-shapes-basics.html - TensorRT 11.4.0.
13. INormalizationLayer (C++ API; Python API Layers page) - https://docs.nvidia.com/deeplearning/tensorrt/10.x.x/_static/c-api/classnvinfer1_1_1_i_normalization_layer.html and https://docs.nvidia.com/deeplearning/tensorrt/10.x.x/_static/python-api/infer/Graph/Layers.html - TensorRT 10.16.1.
14. Working with Transformers (multi-head attention fusion) - https://docs.nvidia.com/deeplearning/tensorrt/10.x.x/inference-library/work-with-transformers.html - TensorRT 10.x docs set (10.16.1).
15. Fused Attention - https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/transformers-fused-attention.html - TensorRT 11.4.0.
16. TensorRT 10.14.1 Release Notes - https://docs.nvidia.com/deeplearning/tensorrt/10.x.x/getting-started/release-notes-10/10.14.1.html - TensorRT 10.14.1.
17. onnx-tensorrt Supported ONNX Operators - https://github.com/onnx/onnx-tensorrt/blob/10.5-GA/docs/operators.md (header reads TensorRT 10.4, opsets 9-20), https://github.com/onnx/onnx-tensorrt/blob/10.11-GA/docs/operators.md (10.11, opsets 9-23), https://github.com/onnx/onnx-tensorrt/blob/main/docs/operators.md (11.3, opsets 9-24).
18. onnx-tensorrt Conv and STFT importers - https://github.com/onnx/onnx-tensorrt/blob/10.5-GA/onnxOpImporters.cpp - branch 10.5-GA.
19. Optimizing Builder Performance - https://docs.nvidia.com/deeplearning/tensorrt/10.x.x/performance/builder-performance.html - TensorRT 10.x docs set (10.16.1).
20. Engine Compatibility - https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/engine-compatibility.html - TensorRT 11.4.0.
21. Polygraphy example, Comparing Frameworks - https://github.com/NVIDIA/TensorRT/blob/release/10.5/tools/Polygraphy/examples/cli/run/01_comparing_frameworks/README.md and the release/11.3 copy - release/10.5 and release/11.3.
22. Polygraphy CreateConfig source - https://github.com/NVIDIA/TensorRT/blob/release/11.3/tools/Polygraphy/polygraphy/backend/trt/config.py - release/11.3.
23. Polygraphy debug precision source - https://github.com/NVIDIA/TensorRT/blob/release/11.3/tools/Polygraphy/polygraphy/tools/debug/subtool/precision.py - release/11.3.
24. microsoft/wavlm-large config.json - https://huggingface.co/microsoft/wavlm-large/blob/main/config.json - main branch.
25. Triton release notes 24.10, 25.06, 26.05 - https://docs.nvidia.com/deeplearning/triton-inference-server/release-notes/rel-24-10.html, .../rel-25-06.html, .../rel-26-05.html - those releases.
26. TensorRT capabilities, Strong Typing vs Weak Typing - https://docs.nvidia.com/deeplearning/tensorrt/10.x.x/architecture/capabilities.html - TensorRT 10.x docs set (10.16.1).
27. PyTorch ONNX exporter, symbolic_opset17.py - https://github.com/pytorch/pytorch/blob/v2.1.1/torch/onnx/symbolic_opset17.py - PyTorch v2.1.1.
28. torchaudio kaldi.py - https://github.com/pytorch/audio/blob/v2.1.1/torchaudio/compliance/kaldi.py - torchaudio v2.1.1.
29. pyannote WeSpeaker wrapper - https://github.com/pyannote/pyannote-audio/blob/3.1.1/pyannote/audio/models/embedding/wespeaker/__init__.py - pyannote-audio 3.1.1.