src/triton_inference/docs/reference/03-dynamic-batching.md

Anuvani-Diarization / src /triton_inference /docs /reference /03-dynamic-batching.md

Anuvani-Diarization 1.0.0

63dd946 12 days ago

11.3 kB

# Dynamic batching, ragged inputs and client-side packing

This note covers what Triton's dynamic batcher needs, what it does to a model with `max_batch_size: 0`, how ragged
inputs and batch inputs work, how the Python backend receives a dynamic batch, and the alternative of packing
blocks in the client. The facts apply to Triton 24.10 (the `r24.10` branches of `server`, `core`, `common` and
`python_backend`); where core `main` or `r25.06` was checked, the text says so. Behaviour read from source code is
marked as such. Checked 9 October 2026.

## What the dynamic batcher requires

- Batching by the dynamic or sequence batcher needs the model's batch dimension to be the first dimension of every
  input and output; `max_batch_size` is then set to 1 or more. Models that do not support batching "in the specific
  ways described above" must set `max_batch_size` to zero [1].
- Dynamic batching combines requests in the server. It is enabled per model with `dynamic_batching { }`, and its
  settings control the preferred batch sizes, how long a request may wait for others, and queue properties such as
  queue size, priorities and time-outs [1].
- "By default, the requests can be dynamically batched only if each input has the same shape across the requests" [2].
  The proto says the same of `allow_ragged_batch`: false means two requests batch only if the tensor has the same
  shape in both [3]. In core, every variable-size input (any `-1`) that is not marked `allow_ragged_batch`, and every
  shape tensor, is put in an "enforce equal shape" set used when forming a batch [4].
- `preferred_batch_size`: "should not be used for most models"; the batcher forms the largest preferred size it can,
  otherwise the largest size up to `max_batch_size` [1]. If none is given, core fills in `max_batch_size`, but only
  when `max_batch_size > 0` [5].
- `max_queue_delay_microseconds` (default 0) lets a batcher hold a batch that is not of a maximum or preferred size
  until the oldest request has waited that long [1, 3].
- `preserve_ordering` returns responses in request order [1, 3]. Validation rejects it together with more than one
  priority level, or with a queue time-out whose action is `DELAY` [5].
- Custom batching rules can be added through five `TRITONBACKEND_ModelBatch*` functions in a shared library named by
  the `TRITON_BATCH_STRATEGY_PATH` parameter [1].
- Without any scheduling choice the model gets the default scheduler, which "simply distributes inference requests to
  all model instances" [1]. In core this is the dynamic batch scheduler with batching switched off; each request goes
  straight to the rate limiter as its own payload [4, 6].

## What happens with `max_batch_size: 0`

- Validation in core r24.10 rejects a negative `max_batch_size` and any `preferred_batch_size` larger than
  `max_batch_size` ("dynamic batching preferred size must be <= max batch size"), but it does not reject a
  `dynamic_batching` block on a model with `max_batch_size: 0` [5].
- When `dynamic_batching` is present, core creates the dynamic batch scheduler with batching enabled and the model's
  `max_batch_size` [4]. The scheduler stores `max_batch_size_((size_t)std::max(1, max_batch_size))`, so a model with
  `max_batch_size: 0` is given an effective limit of one request per execution [6]. The same line is on core `main` [7].
- Read together (source reading, not a documented contract): such a model gets the batcher's queue (priority levels,
  queue policy, time-outs, `max_queue_size`) but never more than one request in an `execute()` call.

## Ragged inputs and batch inputs

- `allow_ragged_batch: true` on an input means two requests may batch even if that tensor's shape differs [3].
- "How ragged input are processed in a batch of requests depends on the backend implementation." The ONNX Runtime,
  TensorFlow, PyTorch and TensorRT backends require ragged inputs to be 1-D and concatenate the requests' inputs into
  one 1-D tensor [2]. The Python backend is not on that list [2].
- Because the concatenated tensor does not record where each request starts and ends, Triton can generate batch
  inputs [2]. The kinds are [3]: `BATCH_ELEMENT_COUNT` (element count of the source input, shape [1]);
  `BATCH_ACCUMULATED_ELEMENT_COUNT` (running total); `BATCH_ACCUMULATED_ELEMENT_COUNT_WITH_ZERO` (the same with a
  leading 0 on the first request); `BATCH_MAX_ELEMENT_COUNT_AS_SHAPE` (shape [max element count], data
  uninitialised); `BATCH_ITEM_SHAPE` and `BATCH_ITEM_SHAPE_FLATTEN` (each request's source shape). The batch input
  data type can be `TYPE_INT32` or `TYPE_FP32` [3].
- Worked example from the docs: requests of shapes `[1, 3]`, `[1, 4]` and `[1, 5]` become one tensor of shape `[12]`,
  and a `BATCH_ACCUMULATED_ELEMENT_COUNT` input named `INDEX` holds `[3, 7, 12]` [2].

## How the Python backend receives a dynamic batch

- `execute()` "receives a batch of pb_utils.InferenceRequest as a length N array" and must return exactly N
  responses, one per request, in the same order [8]. The README's template says "Every Python backend must iterate
  through list of requests" [8]. Each request keeps its own input tensors; nothing is concatenated for the model.
- By default the Python backend moves every input tensor to CPU before the model sees it;
  `FORCE_CPU_ONLY_INPUT_TENSORS: "no"` lets inputs stay where they were last used [8].
- `python_be.cc` on r24.10 counts the batch size per request (first dimension when `max_batch_size > 0`, otherwise 1
  per request) and contains no reference to batch inputs [9]. Inference: batch inputs are not generated for Python
  models, so a Python model has to work out the boundaries from the request list itself.

## Why the batcher cannot pack the diarization requests (inference)

The points below are reasoning from the rules above, not documented statements.

- `diar_nn_g<i>` has `max_batch_size: 0`, so there is no batch dimension, and a `dynamic_batching` block would give it
  a queue but still one request per execution [5, 6].
- Giving the inputs a batch dimension does not help on its own: `SPAN`, `OFFSETS`, `VALID`, `WEIGHTS` and `IDX` vary
  in length, so without `allow_ragged_batch` two requests would batch only when every length matched exactly [2, 4].
- With `allow_ragged_batch`, the Python backend still delivers separate request objects [8]. The model would have
  to group requests by `OP`, concatenate spans and rebase `OFFSETS` itself, which is the packing work in a new place.
- The batcher does not look at input values, so one batch could mix `OP` 0 (segmentation) and `OP` 1 or 2
  (voiceprints), which cannot be one network call. Keeping them apart would need a custom batching library in C [1]
  or one model per `OP`.
- The second call of a block depends on masks the engine computes from the first call, so the two calls of one block
  can never share a batch; any `max_queue_delay_microseconds` would add directly to that dependency chain.

## Client-side packing

The alternative keeps `max_batch_size: 0` and lets the Rust engine put several blocks in one request: one longer
`SPAN`, and `OFFSETS`/`VALID` (and `WEIGHTS`/`IDX` for voiceprints) covering 8 x k windows. Trade-offs (inference):

- The engine already chooses which GPU gets each block, so it can also choose k per request; no server-side delay
  or batching policy is involved, and batch composition is repeatable from the engine's side.
- Each packed request carries about k x 1.7 MB, holds k blocks of results until the whole request finishes, and uses
  more GPU memory per call. Fewer, larger requests reduce per-request HTTP and Python overhead.
- Triton 24.10 has no HTTP size limit. The `--http-max-input-size` option of 25.06 and later (default 64 MiB, about
  38 blocks of 1.74 MB) appears, from source, not to cover binary tensors; see 05-protocols-shared-memory.md.
- A packed request is a new batch shape for the networks. The exact lane asks only for the release's batch shapes,
  so a packed shape belongs to the faster lane unless it is checked against the release first.

## Unverified

- That `dynamic_batching` on a `max_batch_size: 0` model yields one request per execution is read from core source;
  no Triton document states it, so it may change.
- That the Python backend never generates batch inputs is inferred from the absence of any reference in
  `python_be.cc` r24.10; no document says so either way.

## What this means for the diarization model

- The dynamic batcher cannot merge the engine's blocks into larger network calls while the model keeps
  `max_batch_size: 0`, variable-length 1-D inputs and an `OP` switch; packing has to happen in the engine (inference).
- Adding `dynamic_batching { }` without `preferred_batch_size` to `diar_nn_g<i>` would leave `execute()` unchanged (one
  request per call) and would add priority levels, `max_queue_size` and per-request time-outs [5, 6]. With 2
  requests in flight per GPU from the engine, a queue limit mainly protects against other clients.
- A packed request of k blocks is one network call of 8 x k windows; for the faster lane this is the route to larger
  batches, and its accuracy against the exact lane decides whether a packed shape is acceptable (inference).
- Moving the masks into the model (one request per block running both networks) would remove the second transfer of
  the same `SPAN`, which is independent of batching (inference; see 02-ensemble-bls-single-model.md).

## Sources

1. Triton Model Configuration (Maximum Batch Size, Default Scheduler, Dynamic Batcher, Custom Batching) -
   https://github.com/triton-inference-server/server/blob/r24.10/docs/user_guide/model_configuration.md - applies to r24.10.
2. Triton Ragged Batching - https://github.com/triton-inference-server/server/blob/r24.10/docs/user_guide/ragged_batching.md - applies to r24.10.
3. `model_config.proto` (ModelInput.allow_ragged_batch, BatchInput, ModelQueuePolicy, ModelDynamicBatching) -
   https://github.com/triton-inference-server/common/blob/r24.10/protobuf/model_config.proto - applies to r24.10.
4. Core `TritonModel::SetConfiguredScheduler` -
   https://github.com/triton-inference-server/core/blob/r24.10/src/backend_model.cc#L797-L847 - applies to r24.10.
5. Core `NormalizeModelConfig` and `ValidateModelConfig` -
   https://github.com/triton-inference-server/core/blob/r24.10/src/model_config_utils.cc#L734-L745 and
   https://github.com/triton-inference-server/core/blob/r24.10/src/model_config_utils.cc#L1410-L1498 - applies to r24.10.
6. Core dynamic batch scheduler (constructor at line 92; non-batching path at lines 240-251) -
   https://github.com/triton-inference-server/core/blob/r24.10/src/dynamic_batch_scheduler.cc - applies to r24.10.
7. Core dynamic batch scheduler on `main` (GitHub code search hit, commit 00e4e60) -
   https://github.com/triton-inference-server/core/blob/main/src/dynamic_batch_scheduler.cc - applies to main, 9 October 2026.
8. Python backend README (`execute`, Default Mode, Input Tensor Device Placement) -
   https://github.com/triton-inference-server/python_backend/blob/r24.10/README.md - applies to r24.10.
9. Python backend `CheckIncomingRequests` -
   https://github.com/triton-inference-server/python_backend/blob/r24.10/src/python_be.cc#L65-L113 - applies to r24.10.

	# Dynamic batching, ragged inputs and client-side packing

	This note covers what Triton's dynamic batcher needs, what it does to a model with `max_batch_size: 0`, how ragged
	inputs and batch inputs work, how the Python backend receives a dynamic batch, and the alternative of packing
	blocks in the client. The facts apply to Triton 24.10 (the `r24.10` branches of `server`, `core`, `common` and
	`python_backend`); where core `main` or `r25.06` was checked, the text says so. Behaviour read from source code is
	marked as such. Checked 9 October 2026.

	## What the dynamic batcher requires

	- Batching by the dynamic or sequence batcher needs the model's batch dimension to be the first dimension of every
	input and output; `max_batch_size` is then set to 1 or more. Models that do not support batching "in the specific
	ways described above" must set `max_batch_size` to zero [1].
	- Dynamic batching combines requests in the server. It is enabled per model with `dynamic_batching { }`, and its
	settings control the preferred batch sizes, how long a request may wait for others, and queue properties such as
	queue size, priorities and time-outs [1].
	- "By default, the requests can be dynamically batched only if each input has the same shape across the requests" [2].
	The proto says the same of `allow_ragged_batch`: false means two requests batch only if the tensor has the same
	shape in both [3]. In core, every variable-size input (any `-1`) that is not marked `allow_ragged_batch`, and every
	shape tensor, is put in an "enforce equal shape" set used when forming a batch [4].
	- `preferred_batch_size`: "should not be used for most models"; the batcher forms the largest preferred size it can,
	otherwise the largest size up to `max_batch_size` [1]. If none is given, core fills in `max_batch_size`, but only
	when `max_batch_size > 0` [5].
	- `max_queue_delay_microseconds` (default 0) lets a batcher hold a batch that is not of a maximum or preferred size
	until the oldest request has waited that long [1, 3].
	- `preserve_ordering` returns responses in request order [1, 3]. Validation rejects it together with more than one
	priority level, or with a queue time-out whose action is `DELAY` [5].
	- Custom batching rules can be added through five `TRITONBACKEND_ModelBatch*` functions in a shared library named by
	the `TRITON_BATCH_STRATEGY_PATH` parameter [1].
	- Without any scheduling choice the model gets the default scheduler, which "simply distributes inference requests to
	all model instances" [1]. In core this is the dynamic batch scheduler with batching switched off; each request goes
	straight to the rate limiter as its own payload [4, 6].

	## What happens with `max_batch_size: 0`

	- Validation in core r24.10 rejects a negative `max_batch_size` and any `preferred_batch_size` larger than
	`max_batch_size` ("dynamic batching preferred size must be <= max batch size"), but it does not reject a
	`dynamic_batching` block on a model with `max_batch_size: 0` [5].
	- When `dynamic_batching` is present, core creates the dynamic batch scheduler with batching enabled and the model's
	`max_batch_size` [4]. The scheduler stores `max_batch_size_((size_t)std::max(1, max_batch_size))`, so a model with
	`max_batch_size: 0` is given an effective limit of one request per execution [6]. The same line is on core `main` [7].
	- Read together (source reading, not a documented contract): such a model gets the batcher's queue (priority levels,
	queue policy, time-outs, `max_queue_size`) but never more than one request in an `execute()` call.

	## Ragged inputs and batch inputs

	- `allow_ragged_batch: true` on an input means two requests may batch even if that tensor's shape differs [3].
	- "How ragged input are processed in a batch of requests depends on the backend implementation." The ONNX Runtime,
	TensorFlow, PyTorch and TensorRT backends require ragged inputs to be 1-D and concatenate the requests' inputs into
	one 1-D tensor [2]. The Python backend is not on that list [2].
	- Because the concatenated tensor does not record where each request starts and ends, Triton can generate batch
	inputs [2]. The kinds are [3]: `BATCH_ELEMENT_COUNT` (element count of the source input, shape [1]);
	`BATCH_ACCUMULATED_ELEMENT_COUNT` (running total); `BATCH_ACCUMULATED_ELEMENT_COUNT_WITH_ZERO` (the same with a
	leading 0 on the first request); `BATCH_MAX_ELEMENT_COUNT_AS_SHAPE` (shape [max element count], data
	uninitialised); `BATCH_ITEM_SHAPE` and `BATCH_ITEM_SHAPE_FLATTEN` (each request's source shape). The batch input
	data type can be `TYPE_INT32` or `TYPE_FP32` [3].
	- Worked example from the docs: requests of shapes `[1, 3]`, `[1, 4]` and `[1, 5]` become one tensor of shape `[12]`,
	and a `BATCH_ACCUMULATED_ELEMENT_COUNT` input named `INDEX` holds `[3, 7, 12]` [2].

	## How the Python backend receives a dynamic batch

	- `execute()` "receives a batch of pb_utils.InferenceRequest as a length N array" and must return exactly N
	responses, one per request, in the same order [8]. The README's template says "Every Python backend must iterate
	through list of requests" [8]. Each request keeps its own input tensors; nothing is concatenated for the model.
	- By default the Python backend moves every input tensor to CPU before the model sees it;
	`FORCE_CPU_ONLY_INPUT_TENSORS: "no"` lets inputs stay where they were last used [8].
	- `python_be.cc` on r24.10 counts the batch size per request (first dimension when `max_batch_size > 0`, otherwise 1
	per request) and contains no reference to batch inputs [9]. Inference: batch inputs are not generated for Python
	models, so a Python model has to work out the boundaries from the request list itself.

	## Why the batcher cannot pack the diarization requests (inference)

	The points below are reasoning from the rules above, not documented statements.

	- `diar_nn_g<i>` has `max_batch_size: 0`, so there is no batch dimension, and a `dynamic_batching` block would give it
	a queue but still one request per execution [5, 6].
	- Giving the inputs a batch dimension does not help on its own: `SPAN`, `OFFSETS`, `VALID`, `WEIGHTS` and `IDX` vary
	in length, so without `allow_ragged_batch` two requests would batch only when every length matched exactly [2, 4].
	- With `allow_ragged_batch`, the Python backend still delivers separate request objects [8]. The model would have
	to group requests by `OP`, concatenate spans and rebase `OFFSETS` itself, which is the packing work in a new place.
	- The batcher does not look at input values, so one batch could mix `OP` 0 (segmentation) and `OP` 1 or 2
	(voiceprints), which cannot be one network call. Keeping them apart would need a custom batching library in C [1]
	or one model per `OP`.
	- The second call of a block depends on masks the engine computes from the first call, so the two calls of one block
	can never share a batch; any `max_queue_delay_microseconds` would add directly to that dependency chain.

	## Client-side packing

	The alternative keeps `max_batch_size: 0` and lets the Rust engine put several blocks in one request: one longer
	`SPAN`, and `OFFSETS`/`VALID` (and `WEIGHTS`/`IDX` for voiceprints) covering 8 x k windows. Trade-offs (inference):

	- The engine already chooses which GPU gets each block, so it can also choose k per request; no server-side delay
	or batching policy is involved, and batch composition is repeatable from the engine's side.
	- Each packed request carries about k x 1.7 MB, holds k blocks of results until the whole request finishes, and uses
	more GPU memory per call. Fewer, larger requests reduce per-request HTTP and Python overhead.
	- Triton 24.10 has no HTTP size limit. The `--http-max-input-size` option of 25.06 and later (default 64 MiB, about
	38 blocks of 1.74 MB) appears, from source, not to cover binary tensors; see 05-protocols-shared-memory.md.
	- A packed request is a new batch shape for the networks. The exact lane asks only for the release's batch shapes,
	so a packed shape belongs to the faster lane unless it is checked against the release first.

	## Unverified

	- That `dynamic_batching` on a `max_batch_size: 0` model yields one request per execution is read from core source;
	no Triton document states it, so it may change.
	- That the Python backend never generates batch inputs is inferred from the absence of any reference in
	`python_be.cc` r24.10; no document says so either way.

	## What this means for the diarization model

	- The dynamic batcher cannot merge the engine's blocks into larger network calls while the model keeps
	`max_batch_size: 0`, variable-length 1-D inputs and an `OP` switch; packing has to happen in the engine (inference).
	- Adding `dynamic_batching { }` without `preferred_batch_size` to `diar_nn_g<i>` would leave `execute()` unchanged (one
	request per call) and would add priority levels, `max_queue_size` and per-request time-outs [5, 6]. With 2
	requests in flight per GPU from the engine, a queue limit mainly protects against other clients.
	- A packed request of k blocks is one network call of 8 x k windows; for the faster lane this is the route to larger
	batches, and its accuracy against the exact lane decides whether a packed shape is acceptable (inference).
	- Moving the masks into the model (one request per block running both networks) would remove the second transfer of
	the same `SPAN`, which is independent of batching (inference; see 02-ensemble-bls-single-model.md).

	## Sources

	1. Triton Model Configuration (Maximum Batch Size, Default Scheduler, Dynamic Batcher, Custom Batching) -
	https://github.com/triton-inference-server/server/blob/r24.10/docs/user_guide/model_configuration.md - applies to r24.10.
	2. Triton Ragged Batching - https://github.com/triton-inference-server/server/blob/r24.10/docs/user_guide/ragged_batching.md - applies to r24.10.
	3. `model_config.proto` (ModelInput.allow_ragged_batch, BatchInput, ModelQueuePolicy, ModelDynamicBatching) -
	https://github.com/triton-inference-server/common/blob/r24.10/protobuf/model_config.proto - applies to r24.10.
	4. Core `TritonModel::SetConfiguredScheduler` -
	https://github.com/triton-inference-server/core/blob/r24.10/src/backend_model.cc#L797-L847 - applies to r24.10.
	5. Core `NormalizeModelConfig` and `ValidateModelConfig` -
	https://github.com/triton-inference-server/core/blob/r24.10/src/model_config_utils.cc#L734-L745 and
	https://github.com/triton-inference-server/core/blob/r24.10/src/model_config_utils.cc#L1410-L1498 - applies to r24.10.
	6. Core dynamic batch scheduler (constructor at line 92; non-batching path at lines 240-251) -
	https://github.com/triton-inference-server/core/blob/r24.10/src/dynamic_batch_scheduler.cc - applies to r24.10.
	7. Core dynamic batch scheduler on `main` (GitHub code search hit, commit 00e4e60) -
	https://github.com/triton-inference-server/core/blob/main/src/dynamic_batch_scheduler.cc - applies to main, 9 October 2026.
	8. Python backend README (`execute`, Default Mode, Input Tensor Device Placement) -
	https://github.com/triton-inference-server/python_backend/blob/r24.10/README.md - applies to r24.10.
	9. Python backend `CheckIncomingRequests` -
	https://github.com/triton-inference-server/python_backend/blob/r24.10/src/python_be.cc#L65-L113 - applies to r24.10.