src/triton_inference/docs/reference/README.md · CIIL/Anuvani-Diarization at main

Anuvani-Diarization / src /triton_inference /docs /reference /README.md

Anuvani-Diarization 1.0.0

63dd946 12 days ago

13.6 kB

# Reference library: serving the diarization networks on Triton

A compact, sourced reference for serving the Anuvani diarization networks on NVIDIA Triton Inference Server, and
for a faster lane. Each file covers one topic, states the versions its facts apply
to, and ends with what the facts mean for this model. Checked 9 October 2026.

## How to read these files

- Every factual sentence carries a tag such as [3] that points to the file's Sources list, where each source has
  its URL and the version it applies to (a Triton release branch such as `r24.10`, a TensorRT or PyTorch version,
  or a documentation build).
- Anything that could not be confirmed in a primary source is marked "unverified" and listed under "Unverified".
- A source that begins "Contents of the image" records what a published NVIDIA image contains, read from the image
  with the commands it names. It is not NVIDIA documentation.
- The closing section of each file, "What this means for the diarization model", reasons from those facts and the
  model facts below. It is inference, labelled as such; nothing in this library was measured on a GPU.

## The model these files are about (project facts)

- Audio is cut into 16 s windows every 1.6 s (256,000 samples at 16 kHz); windows go in blocks of 8, and the span
  of one block is about 1.7 MB of fp32 samples.
- Per block, two dependent network calls. (1) Segmentation: a pruned WavLM-Large (24 layers, 0 to 10 attention
  heads per layer, four layers without attention, FFN widths 96 to 1770, a 7-layer raw-waveform convolutional
  front end), a 4-layer Conformer and a 16-class powerset head; output (B, 799, 16). (2) Voiceprints: kaldi fbank
  (80 mel, through `torch.vmap`), a WeSpeaker ResNet34 trunk, weighted statistics pooling per local speaker (799
  frame weights interpolated to 200 frames) and a linear layer; output 256-dim.
- Work per window: 146.3 GFLOP (segmentation) plus 72.5 GFLOP (voiceprints). Masks (argmax and an 11-frame median
  filter) are computed between the two calls, in the Rust engine, so each block sends its span twice. Clustering
  (AHC, PLDA, VBx), the tree decoder and the RTTM writer run on the CPU in Rust.
- Serving today: Triton 24.10; Python backend models `diar_nn_g<i>`, one per GPU, `max_batch_size: 0`, one
  instance, one CUDA stream; inputs OP int32 (0 segmentation, 1 voiceprints, 2 repacked voiceprints), SPAN fp32,
  OFFSETS and VALID int64, optional WEIGHTS fp32 and IDX int64; output OUT fp32. PyTorch 2.1.1 in FP32 with TF32
  off, byte-identical to the frozen release.
- The Rust engine is Triton's only client (KServe v2 HTTP with binary tensors), chooses the GPU for each block and
  keeps 2 requests in flight per GPU; a FastAPI service will sit in front of it. These notes call the present FP32
  path the exact lane. A faster lane (TF32, BF16 or FP16; TensorRT or torch.compile; several blocks per request;
  two CUDA streams) is allowed if a DER accuracy gate passes.

## Files

| File | What it covers |
|---|---|
| [01-python-backend.md](01-python-backend.md) | How `execute()` is called; one stub process and CUDA context per instance; the shared-memory data path; DLPack, BLS, decoupled mode, custom Python environments; a TensorRT engine on torch's stream |
| [02-ensemble-bls-single-model.md](02-ensemble-bls-single-model.md) | Ensemble, BLS or one Python model for "network, glue, network", and the cheapest server-side form for this model |
| [03-dynamic-batching.md](03-dynamic-batching.md) | What the dynamic batcher needs, what it does with `max_batch_size: 0`, ragged inputs, how a Python model sees a batch, client-side packing |
| [04-instances-streams-mps.md](04-instances-streams-mps.md) | Instance groups, streams in one process, time-slicing between contexts, TensorRT-backend stream options, CUDA MPS and its limits |
| [05-protocols-shared-memory.md](05-protocols-shared-memory.md) | HTTP against gRPC, binary tensors, compression, message size limits, system and CUDA shared memory, the Python backend's own copy |
| [06-tensorrt-build.md](06-tensorrt-build.md) | TensorRT 10 against 11 typing, precision by compute capability, profiles, attention fusion, LayerNorm, ONNX operators for the front ends, build settings, portability, Polygraphy |
| [07-reduced-precision-hazards.md](07-reduced-precision-hazards.md) | FP16, BF16 and TF32 ranges, reductions, autocast lists, pre-LN residual growth, fbank values near 1e14, epsilons, FP32 islands |
| [08-torch-compile-aotinductor.md](08-torch-compile-aotinductor.md) | torch.compile modes, recompilation and caches (2.1 against 2.14), AOTInductor, Triton's PyTorch backend and its libtorch |
| [09-cuda-graphs.md](09-cuda-graphs.md) | Capture rules in CUDA, PyTorch, TensorRT and Triton, and when graphs pay off |
| [10-cutlass-cublaslt.md](10-cutlass-cublaslt.md) | cuBLASLt epilogues, cuBLAS compute types and reproducibility, what PyTorch 2.1.1 already sends to cuBLASLt, CUTLASS, custom kernels |
| [11-profiling-mfu.md](11-profiling-mfu.md) | Nsight Systems, Nsight Compute, trtexec profiling, DCGM profiling fields, MFU and HFU |
| [12-perf-analyzer-model-analyzer.md](12-perf-analyzer-model-analyzer.md) | perf_analyzer input data and search options, Model Analyzer, builds that run on glibc 2.35 |
| [13-server-operations.md](13-server-operations.md) | Metrics (queue against compute time), rate limiter, priority and queue policy, response cache, model control, health, shutdown |
| [14-tensorrt-llm.md](14-tensorrt-llm.md) | What TensorRT-LLM supports, its removed plugins, how Triton packages it, and why it does not fit this pipeline |
| [15-triton-image-versions.md](15-triton-image-versions.md) | Python, CUDA, TensorRT and driver needs of the 24.10, 25.06 and 26.05 images |
## Findings that cut across files
- The exact lane stays on Triton 24.10. From 24.11 the images are Ubuntu 24.04 with Python 3.12 [1], and torch 2.1.1
  wheels exist only for CPython 3.8 to 3.11 [2]. 26.05 is built on CUDA 13.2.1, which needs a driver of 580 or
  later [3]; on an older driver CUDA fails to initialise with CUDA_ERROR_SYSTEM_DRIVER_MISMATCH unless forward
  compatibility applies [25]. (01, 15)
- Triton's PyTorch backend would change the numerics: in 24.10 it is built from the NGC PyTorch 24.10 container,
  PyTorch 2.5.0a0+e000cf0ad9 with cuBLAS 12.6.3.3 [4, 5]. (08)
- The dynamic batcher cannot make the network calls larger. A `dynamic_batching` block on a `max_batch_size: 0`
  model is accepted, but the scheduler caps each execution at max(1, `max_batch_size`) requests [6]; it adds a
  queue (priorities, queue size, time-outs) and no batching. Larger calls have to be packed by the Rust engine.
  (03, 13)
- One request per block is the structural saving in sight: a third OP that runs segmentation, masks and
  voiceprints in one `execute()` would send the span once instead of twice and remove a round trip, keeping one
  process, one context and one stream. It is exact only if the PyTorch masks equal the Rust masks bit for bit
  (inference). Ensembles and BLS would add models, processes and internal requests. (02, 05)
- On transport, 24.10 has no option for an HTTP body limit; `--http-max-input-size` (default 64 MiB) arrives in
  25.06 [7] and, from the source, appears not to cover binary tensors. The Python backend copies every CPU input
  into its own shared-memory pool [8], so client shared memory would save the HTTP body but not that copy. No NVIDIA
  document compares HTTP and gRPC at this request size. (05)
- Kernels from different CUDA contexts, such as two Python instances, do not run concurrently; the GPU time-slices
  between them unless MPS is used [9]. cuBLAS promises bit-wise results only while a single stream is active [10],
  which matches the release's one-stream rule; two streams belong to the gated lane. (04, 10)
- TF32 off needs both PyTorch switches, because `cudnn.allow_tf32` defaults to True [11]; the release sets both.
  TensorRT enables TF32 by default [12], while Polygraphy disables it by default [13]. (06, 07, 10)
- TensorRT 11 removed weak typing [14]; precision then follows the tensor types, and FP32 islands are written as
  casts around the operations that must stay FP32 [23]. The three images carry TensorRT 10.5, 10.11 and 10.16 (see
  15). Fused attention is documented for FP16, BF16, INT8 and FP8, with no FP32 or TF32 entry [15], and the
  onnx-tensorrt operator list for 10.5 marks `DFT` unsupported [16], so the fbank stays in PyTorch FP32. FP16 tops
  out at 65,504 [12], far below the fbank's powers of about 1e14. (06, 07)
- PyTorch does not promise bitwise-identical results for mathematically identical floating-point computations [24],
  and no document promises that compiled output equals eager output. 2.1.1 recompiles a frame up to 64 times before
  falling back to eager [17]. TensorRT puts a kernel launch at 5 to 15 µs of host time [18]; when a call's GPU work
  is far longer than its launch time, CUDA graphs help only if Nsight Systems shows the GPU waiting (inference).
  (08, 09)
- The current Nsight Compute needs a CUDA 13 driver [19], so a driver older than R580 needs an older release;
  DCGM profiling must be paused around Nsight runs [20]. On a glibc 2.35 host (Ubuntu 22.04), perf_analyzer runs
  from the 24.01 SDK image or tritonclient 2.51.0; the tritonclient wheels from 2.52.0 and the perf-analyzer wheels
  need GLIBC_2.38 [21]. (11, 12)
- TensorRT-LLM does not fit: from release 1.2 it has no TensorRT engine backend [22], and its documented models
  do not include this kind of pipeline. (14)

## Main points left unverified (details in each file)

- One request per `execute()` with `dynamic_batching` and `max_batch_size: 0` comes from core source, not from a
  document (03).
- That `--http-max-input-size` does not cover binary tensors comes from reading the source; it was not tested (05).
- Which RTX GPUs are "NGC Server Ready" SKUs eligible for forward compatibility, and whether DCGM profiling is
  supported on workstation GPUs (15, 11).
- Whether compiled, graph-captured or TensorRT FP32 output can ever equal the release's eager bytes (06, 08, 09).
- Whether PyTorch 2.1.1 can export `torch.fft.rfft` under `torch.vmap` to ONNX, and whether WavLM's gated relative
  position bias still matches TensorRT's attention fusion pattern (06).
- Which Nsight Compute release is the newest that runs on a driver older than R580 (11).

## Sources

1. Triton Release 24.11 - https://docs.nvidia.com/deeplearning/triton-inference-server/release-notes/rel-24-11.html - 24.11.
2. torch 2.1.1 files - https://pypi.org/project/torch/2.1.1/#files - checked 9 October 2026.
3. CUDA Toolkit Release Notes, Tables 2 and 3 - https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html - CUDA 13.4 Update 1.
4. `build.py` - https://github.com/triton-inference-server/server/blob/r24.10/build.py - r24.10.
5. PyTorch Release 24.10 - https://docs.nvidia.com/deeplearning/frameworks/pytorch-release-notes/rel-24-10.html - NGC PyTorch 24.10.
6. `dynamic_batch_scheduler.cc` - https://github.com/triton-inference-server/core/blob/r24.10/src/dynamic_batch_scheduler.cc - r24.10.
7. `command_line_parser.cc` - https://github.com/triton-inference-server/server/blob/r25.06/src/command_line_parser.cc - r25.06.
8. `python_be.cc` - https://github.com/triton-inference-server/python_backend/blob/r24.10/src/python_be.cc - r24.10.
9. CUDA C++ Programming Guide, 3.2.8 - https://docs.nvidia.com/cuda/archive/12.6.2/cuda-c-programming-guide/index.html - CUDA 12.6.2.
10. cuBLAS, Results reproducibility - https://docs.nvidia.com/cuda/archive/12.1.0/cublas/index.html - cuBLAS 12.1.
11. CUDA semantics, TF32 on Ampere - https://docs.pytorch.org/docs/2.1/notes/cuda.html - PyTorch 2.1.
12. Accuracy Considerations - https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/accuracy-considerations.html - TensorRT 11.4.0.
13. Polygraphy `CreateConfig` - https://github.com/NVIDIA/TensorRT/blob/release/11.3/tools/Polygraphy/polygraphy/backend/trt/config.py - release/11.3.
14. TensorRT 11.0.0 Release Notes - https://docs.nvidia.com/deeplearning/tensorrt/latest/getting-started/release-notes-11/11.0.0.html - TensorRT 11.0.0.
15. Fused Attention - https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/transformers-fused-attention.html - TensorRT 11.4.0.
16. Supported ONNX Operators - https://github.com/onnx/onnx-tensorrt/blob/10.5-GA/docs/operators.md - onnx-tensorrt 10.5-GA.
17. `torch/_dynamo/config.py` - https://github.com/pytorch/pytorch/blob/v2.1.1/torch/_dynamo/config.py - v2.1.1.
18. Optimizing TensorRT Performance, CUDA Graphs - https://docs.nvidia.com/deeplearning/tensorrt/latest/performance/optimization.html - TensorRT 11.4.0.
19. Nsight Compute Release Notes - https://docs.nvidia.com/nsight-compute/ReleaseNotes/index.html - v2026.3.1.
20. DCGM Feature Overview, Profiling Metrics - https://docs.nvidia.com/datacenter/dcgm/latest/user-guide/feature-overview.html - DCGM 4.7.0 docs.
21. tritonclient release files - https://pypi.org/project/tritonclient/#history - 2.51.0 to 2.73.0.
22. TensorRT-LLM release notes, Release 1.2 - https://nvidia.github.io/TensorRT-LLM/release-notes.html - docs 1.3.0rc29.
23. Precision Control - https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/precision-control.html - TensorRT 11.4.0.
24. Numerical accuracy - https://docs.pytorch.org/docs/2.14/notes/numerical_accuracy.html - PyTorch 2.14.
25. CUDA Compatibility, Forward Compatibility - https://docs.nvidia.com/deploy/cuda-compatibility/forward-compatibility.html - updated 9 September 2026.

	# Reference library: serving the diarization networks on Triton

	A compact, sourced reference for serving the Anuvani diarization networks on NVIDIA Triton Inference Server, and
	for a faster lane. Each file covers one topic, states the versions its facts apply
	to, and ends with what the facts mean for this model. Checked 9 October 2026.

	## How to read these files

	- Every factual sentence carries a tag such as [3] that points to the file's Sources list, where each source has
	its URL and the version it applies to (a Triton release branch such as `r24.10`, a TensorRT or PyTorch version,
	or a documentation build).
	- Anything that could not be confirmed in a primary source is marked "unverified" and listed under "Unverified".
	- A source that begins "Contents of the image" records what a published NVIDIA image contains, read from the image
	with the commands it names. It is not NVIDIA documentation.
	- The closing section of each file, "What this means for the diarization model", reasons from those facts and the
	model facts below. It is inference, labelled as such; nothing in this library was measured on a GPU.

	## The model these files are about (project facts)

	- Audio is cut into 16 s windows every 1.6 s (256,000 samples at 16 kHz); windows go in blocks of 8, and the span
	of one block is about 1.7 MB of fp32 samples.
	- Per block, two dependent network calls. (1) Segmentation: a pruned WavLM-Large (24 layers, 0 to 10 attention
	heads per layer, four layers without attention, FFN widths 96 to 1770, a 7-layer raw-waveform convolutional
	front end), a 4-layer Conformer and a 16-class powerset head; output (B, 799, 16). (2) Voiceprints: kaldi fbank
	(80 mel, through `torch.vmap`), a WeSpeaker ResNet34 trunk, weighted statistics pooling per local speaker (799
	frame weights interpolated to 200 frames) and a linear layer; output 256-dim.
	- Work per window: 146.3 GFLOP (segmentation) plus 72.5 GFLOP (voiceprints). Masks (argmax and an 11-frame median
	filter) are computed between the two calls, in the Rust engine, so each block sends its span twice. Clustering
	(AHC, PLDA, VBx), the tree decoder and the RTTM writer run on the CPU in Rust.
	- Serving today: Triton 24.10; Python backend models `diar_nn_g<i>`, one per GPU, `max_batch_size: 0`, one
	instance, one CUDA stream; inputs OP int32 (0 segmentation, 1 voiceprints, 2 repacked voiceprints), SPAN fp32,
	OFFSETS and VALID int64, optional WEIGHTS fp32 and IDX int64; output OUT fp32. PyTorch 2.1.1 in FP32 with TF32
	off, byte-identical to the frozen release.
	- The Rust engine is Triton's only client (KServe v2 HTTP with binary tensors), chooses the GPU for each block and
	keeps 2 requests in flight per GPU; a FastAPI service will sit in front of it. These notes call the present FP32
	path the exact lane. A faster lane (TF32, BF16 or FP16; TensorRT or torch.compile; several blocks per request;
	two CUDA streams) is allowed if a DER accuracy gate passes.

	## Files

	\| File \| What it covers \|
	\|---\|---\|
	\| [01-python-backend.md](01-python-backend.md) \| How `execute()` is called; one stub process and CUDA context per instance; the shared-memory data path; DLPack, BLS, decoupled mode, custom Python environments; a TensorRT engine on torch's stream \|
	\| [02-ensemble-bls-single-model.md](02-ensemble-bls-single-model.md) \| Ensemble, BLS or one Python model for "network, glue, network", and the cheapest server-side form for this model \|
	\| [03-dynamic-batching.md](03-dynamic-batching.md) \| What the dynamic batcher needs, what it does with `max_batch_size: 0`, ragged inputs, how a Python model sees a batch, client-side packing \|
	\| [04-instances-streams-mps.md](04-instances-streams-mps.md) \| Instance groups, streams in one process, time-slicing between contexts, TensorRT-backend stream options, CUDA MPS and its limits \|
	\| [05-protocols-shared-memory.md](05-protocols-shared-memory.md) \| HTTP against gRPC, binary tensors, compression, message size limits, system and CUDA shared memory, the Python backend's own copy \|
	\| [06-tensorrt-build.md](06-tensorrt-build.md) \| TensorRT 10 against 11 typing, precision by compute capability, profiles, attention fusion, LayerNorm, ONNX operators for the front ends, build settings, portability, Polygraphy \|
	\| [07-reduced-precision-hazards.md](07-reduced-precision-hazards.md) \| FP16, BF16 and TF32 ranges, reductions, autocast lists, pre-LN residual growth, fbank values near 1e14, epsilons, FP32 islands \|
	\| [08-torch-compile-aotinductor.md](08-torch-compile-aotinductor.md) \| torch.compile modes, recompilation and caches (2.1 against 2.14), AOTInductor, Triton's PyTorch backend and its libtorch \|
	\| [09-cuda-graphs.md](09-cuda-graphs.md) \| Capture rules in CUDA, PyTorch, TensorRT and Triton, and when graphs pay off \|
	\| [10-cutlass-cublaslt.md](10-cutlass-cublaslt.md) \| cuBLASLt epilogues, cuBLAS compute types and reproducibility, what PyTorch 2.1.1 already sends to cuBLASLt, CUTLASS, custom kernels \|
	\| [11-profiling-mfu.md](11-profiling-mfu.md) \| Nsight Systems, Nsight Compute, trtexec profiling, DCGM profiling fields, MFU and HFU \|
	\| [12-perf-analyzer-model-analyzer.md](12-perf-analyzer-model-analyzer.md) \| perf_analyzer input data and search options, Model Analyzer, builds that run on glibc 2.35 \|
	\| [13-server-operations.md](13-server-operations.md) \| Metrics (queue against compute time), rate limiter, priority and queue policy, response cache, model control, health, shutdown \|
	\| [14-tensorrt-llm.md](14-tensorrt-llm.md) \| What TensorRT-LLM supports, its removed plugins, how Triton packages it, and why it does not fit this pipeline \|
	\| [15-triton-image-versions.md](15-triton-image-versions.md) \| Python, CUDA, TensorRT and driver needs of the 24.10, 25.06 and 26.05 images \|

	## Findings that cut across files

	- The exact lane stays on Triton 24.10. From 24.11 the images are Ubuntu 24.04 with Python 3.12 [1], and torch 2.1.1
	wheels exist only for CPython 3.8 to 3.11 [2]. 26.05 is built on CUDA 13.2.1, which needs a driver of 580 or
	later [3]; on an older driver CUDA fails to initialise with CUDA_ERROR_SYSTEM_DRIVER_MISMATCH unless forward
	compatibility applies [25]. (01, 15)
	- Triton's PyTorch backend would change the numerics: in 24.10 it is built from the NGC PyTorch 24.10 container,
	PyTorch 2.5.0a0+e000cf0ad9 with cuBLAS 12.6.3.3 [4, 5]. (08)
	- The dynamic batcher cannot make the network calls larger. A `dynamic_batching` block on a `max_batch_size: 0`
	model is accepted, but the scheduler caps each execution at max(1, `max_batch_size`) requests [6]; it adds a
	queue (priorities, queue size, time-outs) and no batching. Larger calls have to be packed by the Rust engine.
	(03, 13)
	- One request per block is the structural saving in sight: a third OP that runs segmentation, masks and
	voiceprints in one `execute()` would send the span once instead of twice and remove a round trip, keeping one
	process, one context and one stream. It is exact only if the PyTorch masks equal the Rust masks bit for bit
	(inference). Ensembles and BLS would add models, processes and internal requests. (02, 05)
	- On transport, 24.10 has no option for an HTTP body limit; `--http-max-input-size` (default 64 MiB) arrives in
	25.06 [7] and, from the source, appears not to cover binary tensors. The Python backend copies every CPU input
	into its own shared-memory pool [8], so client shared memory would save the HTTP body but not that copy. No NVIDIA
	document compares HTTP and gRPC at this request size. (05)
	- Kernels from different CUDA contexts, such as two Python instances, do not run concurrently; the GPU time-slices
	between them unless MPS is used [9]. cuBLAS promises bit-wise results only while a single stream is active [10],
	which matches the release's one-stream rule; two streams belong to the gated lane. (04, 10)
	- TF32 off needs both PyTorch switches, because `cudnn.allow_tf32` defaults to True [11]; the release sets both.
	TensorRT enables TF32 by default [12], while Polygraphy disables it by default [13]. (06, 07, 10)
	- TensorRT 11 removed weak typing [14]; precision then follows the tensor types, and FP32 islands are written as
	casts around the operations that must stay FP32 [23]. The three images carry TensorRT 10.5, 10.11 and 10.16 (see
	15). Fused attention is documented for FP16, BF16, INT8 and FP8, with no FP32 or TF32 entry [15], and the
	onnx-tensorrt operator list for 10.5 marks `DFT` unsupported [16], so the fbank stays in PyTorch FP32. FP16 tops
	out at 65,504 [12], far below the fbank's powers of about 1e14. (06, 07)
	- PyTorch does not promise bitwise-identical results for mathematically identical floating-point computations [24],
	and no document promises that compiled output equals eager output. 2.1.1 recompiles a frame up to 64 times before
	falling back to eager [17]. TensorRT puts a kernel launch at 5 to 15 µs of host time [18]; when a call's GPU work
	is far longer than its launch time, CUDA graphs help only if Nsight Systems shows the GPU waiting (inference).
	(08, 09)
	- The current Nsight Compute needs a CUDA 13 driver [19], so a driver older than R580 needs an older release;
	DCGM profiling must be paused around Nsight runs [20]. On a glibc 2.35 host (Ubuntu 22.04), perf_analyzer runs
	from the 24.01 SDK image or tritonclient 2.51.0; the tritonclient wheels from 2.52.0 and the perf-analyzer wheels
	need GLIBC_2.38 [21]. (11, 12)
	- TensorRT-LLM does not fit: from release 1.2 it has no TensorRT engine backend [22], and its documented models
	do not include this kind of pipeline. (14)

	## Main points left unverified (details in each file)

	- One request per `execute()` with `dynamic_batching` and `max_batch_size: 0` comes from core source, not from a
	document (03).
	- That `--http-max-input-size` does not cover binary tensors comes from reading the source; it was not tested (05).
	- Which RTX GPUs are "NGC Server Ready" SKUs eligible for forward compatibility, and whether DCGM profiling is
	supported on workstation GPUs (15, 11).
	- Whether compiled, graph-captured or TensorRT FP32 output can ever equal the release's eager bytes (06, 08, 09).
	- Whether PyTorch 2.1.1 can export `torch.fft.rfft` under `torch.vmap` to ONNX, and whether WavLM's gated relative
	position bias still matches TensorRT's attention fusion pattern (06).
	- Which Nsight Compute release is the newest that runs on a driver older than R580 (11).

	## Sources

	1. Triton Release 24.11 - https://docs.nvidia.com/deeplearning/triton-inference-server/release-notes/rel-24-11.html - 24.11.
	2. torch 2.1.1 files - https://pypi.org/project/torch/2.1.1/#files - checked 9 October 2026.
	3. CUDA Toolkit Release Notes, Tables 2 and 3 - https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html - CUDA 13.4 Update 1.
	4. `build.py` - https://github.com/triton-inference-server/server/blob/r24.10/build.py - r24.10.
	5. PyTorch Release 24.10 - https://docs.nvidia.com/deeplearning/frameworks/pytorch-release-notes/rel-24-10.html - NGC PyTorch 24.10.
	6. `dynamic_batch_scheduler.cc` - https://github.com/triton-inference-server/core/blob/r24.10/src/dynamic_batch_scheduler.cc - r24.10.
	7. `command_line_parser.cc` - https://github.com/triton-inference-server/server/blob/r25.06/src/command_line_parser.cc - r25.06.
	8. `python_be.cc` - https://github.com/triton-inference-server/python_backend/blob/r24.10/src/python_be.cc - r24.10.
	9. CUDA C++ Programming Guide, 3.2.8 - https://docs.nvidia.com/cuda/archive/12.6.2/cuda-c-programming-guide/index.html - CUDA 12.6.2.
	10. cuBLAS, Results reproducibility - https://docs.nvidia.com/cuda/archive/12.1.0/cublas/index.html - cuBLAS 12.1.
	11. CUDA semantics, TF32 on Ampere - https://docs.pytorch.org/docs/2.1/notes/cuda.html - PyTorch 2.1.
	12. Accuracy Considerations - https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/accuracy-considerations.html - TensorRT 11.4.0.
	13. Polygraphy `CreateConfig` - https://github.com/NVIDIA/TensorRT/blob/release/11.3/tools/Polygraphy/polygraphy/backend/trt/config.py - release/11.3.
	14. TensorRT 11.0.0 Release Notes - https://docs.nvidia.com/deeplearning/tensorrt/latest/getting-started/release-notes-11/11.0.0.html - TensorRT 11.0.0.
	15. Fused Attention - https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/transformers-fused-attention.html - TensorRT 11.4.0.
	16. Supported ONNX Operators - https://github.com/onnx/onnx-tensorrt/blob/10.5-GA/docs/operators.md - onnx-tensorrt 10.5-GA.
	17. `torch/_dynamo/config.py` - https://github.com/pytorch/pytorch/blob/v2.1.1/torch/_dynamo/config.py - v2.1.1.
	18. Optimizing TensorRT Performance, CUDA Graphs - https://docs.nvidia.com/deeplearning/tensorrt/latest/performance/optimization.html - TensorRT 11.4.0.
	19. Nsight Compute Release Notes - https://docs.nvidia.com/nsight-compute/ReleaseNotes/index.html - v2026.3.1.
	20. DCGM Feature Overview, Profiling Metrics - https://docs.nvidia.com/datacenter/dcgm/latest/user-guide/feature-overview.html - DCGM 4.7.0 docs.
	21. tritonclient release files - https://pypi.org/project/tritonclient/#history - 2.51.0 to 2.73.0.
	22. TensorRT-LLM release notes, Release 1.2 - https://nvidia.github.io/TensorRT-LLM/release-notes.html - docs 1.3.0rc29.
	23. Precision Control - https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/precision-control.html - TensorRT 11.4.0.
	24. Numerical accuracy - https://docs.pytorch.org/docs/2.14/notes/numerical_accuracy.html - PyTorch 2.14.
	25. CUDA Compatibility, Forward Compatibility - https://docs.nvidia.com/deploy/cuda-compatibility/forward-compatibility.html - updated 9 September 2026.