# Dynamic batching, ragged inputs and client-side packing This note covers what Triton's dynamic batcher needs, what it does to a model with `max_batch_size: 0`, how ragged inputs and batch inputs work, how the Python backend receives a dynamic batch, and the alternative of packing blocks in the client. The facts apply to Triton 24.10 (the `r24.10` branches of `server`, `core`, `common` and `python_backend`); where core `main` or `r25.06` was checked, the text says so. Behaviour read from source code is marked as such. Checked 9 October 2026. ## What the dynamic batcher requires - Batching by the dynamic or sequence batcher needs the model's batch dimension to be the first dimension of every input and output; `max_batch_size` is then set to 1 or more. Models that do not support batching "in the specific ways described above" must set `max_batch_size` to zero [1]. - Dynamic batching combines requests in the server. It is enabled per model with `dynamic_batching { }`, and its settings control the preferred batch sizes, how long a request may wait for others, and queue properties such as queue size, priorities and time-outs [1]. - "By default, the requests can be dynamically batched only if each input has the same shape across the requests" [2]. The proto says the same of `allow_ragged_batch`: false means two requests batch only if the tensor has the same shape in both [3]. In core, every variable-size input (any `-1`) that is not marked `allow_ragged_batch`, and every shape tensor, is put in an "enforce equal shape" set used when forming a batch [4]. - `preferred_batch_size`: "should not be used for most models"; the batcher forms the largest preferred size it can, otherwise the largest size up to `max_batch_size` [1]. If none is given, core fills in `max_batch_size`, but only when `max_batch_size > 0` [5]. - `max_queue_delay_microseconds` (default 0) lets a batcher hold a batch that is not of a maximum or preferred size until the oldest request has waited that long [1, 3]. - `preserve_ordering` returns responses in request order [1, 3]. Validation rejects it together with more than one priority level, or with a queue time-out whose action is `DELAY` [5]. - Custom batching rules can be added through five `TRITONBACKEND_ModelBatch*` functions in a shared library named by the `TRITON_BATCH_STRATEGY_PATH` parameter [1]. - Without any scheduling choice the model gets the default scheduler, which "simply distributes inference requests to all model instances" [1]. In core this is the dynamic batch scheduler with batching switched off; each request goes straight to the rate limiter as its own payload [4, 6]. ## What happens with `max_batch_size: 0` - Validation in core r24.10 rejects a negative `max_batch_size` and any `preferred_batch_size` larger than `max_batch_size` ("dynamic batching preferred size must be <= max batch size"), but it does not reject a `dynamic_batching` block on a model with `max_batch_size: 0` [5]. - When `dynamic_batching` is present, core creates the dynamic batch scheduler with batching enabled and the model's `max_batch_size` [4]. The scheduler stores `max_batch_size_((size_t)std::max(1, max_batch_size))`, so a model with `max_batch_size: 0` is given an effective limit of one request per execution [6]. The same line is on core `main` [7]. - Read together (source reading, not a documented contract): such a model gets the batcher's queue (priority levels, queue policy, time-outs, `max_queue_size`) but never more than one request in an `execute()` call. ## Ragged inputs and batch inputs - `allow_ragged_batch: true` on an input means two requests may batch even if that tensor's shape differs [3]. - "How ragged input are processed in a batch of requests depends on the backend implementation." The ONNX Runtime, TensorFlow, PyTorch and TensorRT backends require ragged inputs to be 1-D and concatenate the requests' inputs into one 1-D tensor [2]. The Python backend is not on that list [2]. - Because the concatenated tensor does not record where each request starts and ends, Triton can generate batch inputs [2]. The kinds are [3]: `BATCH_ELEMENT_COUNT` (element count of the source input, shape [1]); `BATCH_ACCUMULATED_ELEMENT_COUNT` (running total); `BATCH_ACCUMULATED_ELEMENT_COUNT_WITH_ZERO` (the same with a leading 0 on the first request); `BATCH_MAX_ELEMENT_COUNT_AS_SHAPE` (shape [max element count], data uninitialised); `BATCH_ITEM_SHAPE` and `BATCH_ITEM_SHAPE_FLATTEN` (each request's source shape). The batch input data type can be `TYPE_INT32` or `TYPE_FP32` [3]. - Worked example from the docs: requests of shapes `[1, 3]`, `[1, 4]` and `[1, 5]` become one tensor of shape `[12]`, and a `BATCH_ACCUMULATED_ELEMENT_COUNT` input named `INDEX` holds `[3, 7, 12]` [2]. ## How the Python backend receives a dynamic batch - `execute()` "receives a batch of pb_utils.InferenceRequest as a length N array" and must return exactly N responses, one per request, in the same order [8]. The README's template says "Every Python backend must iterate through list of requests" [8]. Each request keeps its own input tensors; nothing is concatenated for the model. - By default the Python backend moves every input tensor to CPU before the model sees it; `FORCE_CPU_ONLY_INPUT_TENSORS: "no"` lets inputs stay where they were last used [8]. - `python_be.cc` on r24.10 counts the batch size per request (first dimension when `max_batch_size > 0`, otherwise 1 per request) and contains no reference to batch inputs [9]. Inference: batch inputs are not generated for Python models, so a Python model has to work out the boundaries from the request list itself. ## Why the batcher cannot pack the diarization requests (inference) The points below are reasoning from the rules above, not documented statements. - `diar_nn_g` has `max_batch_size: 0`, so there is no batch dimension, and a `dynamic_batching` block would give it a queue but still one request per execution [5, 6]. - Giving the inputs a batch dimension does not help on its own: `SPAN`, `OFFSETS`, `VALID`, `WEIGHTS` and `IDX` vary in length, so without `allow_ragged_batch` two requests would batch only when every length matched exactly [2, 4]. - With `allow_ragged_batch`, the Python backend still delivers separate request objects [8]. The model would have to group requests by `OP`, concatenate spans and rebase `OFFSETS` itself, which is the packing work in a new place. - The batcher does not look at input values, so one batch could mix `OP` 0 (segmentation) and `OP` 1 or 2 (voiceprints), which cannot be one network call. Keeping them apart would need a custom batching library in C [1] or one model per `OP`. - The second call of a block depends on masks the engine computes from the first call, so the two calls of one block can never share a batch; any `max_queue_delay_microseconds` would add directly to that dependency chain. ## Client-side packing The alternative keeps `max_batch_size: 0` and lets the Rust engine put several blocks in one request: one longer `SPAN`, and `OFFSETS`/`VALID` (and `WEIGHTS`/`IDX` for voiceprints) covering 8 x k windows. Trade-offs (inference): - The engine already chooses which GPU gets each block, so it can also choose k per request; no server-side delay or batching policy is involved, and batch composition is repeatable from the engine's side. - Each packed request carries about k x 1.7 MB, holds k blocks of results until the whole request finishes, and uses more GPU memory per call. Fewer, larger requests reduce per-request HTTP and Python overhead. - Triton 24.10 has no HTTP size limit. The `--http-max-input-size` option of 25.06 and later (default 64 MiB, about 38 blocks of 1.74 MB) appears, from source, not to cover binary tensors; see 05-protocols-shared-memory.md. - A packed request is a new batch shape for the networks. The exact lane asks only for the release's batch shapes, so a packed shape belongs to the faster lane unless it is checked against the release first. ## Unverified - That `dynamic_batching` on a `max_batch_size: 0` model yields one request per execution is read from core source; no Triton document states it, so it may change. - That the Python backend never generates batch inputs is inferred from the absence of any reference in `python_be.cc` r24.10; no document says so either way. ## What this means for the diarization model - The dynamic batcher cannot merge the engine's blocks into larger network calls while the model keeps `max_batch_size: 0`, variable-length 1-D inputs and an `OP` switch; packing has to happen in the engine (inference). - Adding `dynamic_batching { }` without `preferred_batch_size` to `diar_nn_g` would leave `execute()` unchanged (one request per call) and would add priority levels, `max_queue_size` and per-request time-outs [5, 6]. With 2 requests in flight per GPU from the engine, a queue limit mainly protects against other clients. - A packed request of k blocks is one network call of 8 x k windows; for the faster lane this is the route to larger batches, and its accuracy against the exact lane decides whether a packed shape is acceptable (inference). - Moving the masks into the model (one request per block running both networks) would remove the second transfer of the same `SPAN`, which is independent of batching (inference; see 02-ensemble-bls-single-model.md). ## Sources 1. Triton Model Configuration (Maximum Batch Size, Default Scheduler, Dynamic Batcher, Custom Batching) - https://github.com/triton-inference-server/server/blob/r24.10/docs/user_guide/model_configuration.md - applies to r24.10. 2. Triton Ragged Batching - https://github.com/triton-inference-server/server/blob/r24.10/docs/user_guide/ragged_batching.md - applies to r24.10. 3. `model_config.proto` (ModelInput.allow_ragged_batch, BatchInput, ModelQueuePolicy, ModelDynamicBatching) - https://github.com/triton-inference-server/common/blob/r24.10/protobuf/model_config.proto - applies to r24.10. 4. Core `TritonModel::SetConfiguredScheduler` - https://github.com/triton-inference-server/core/blob/r24.10/src/backend_model.cc#L797-L847 - applies to r24.10. 5. Core `NormalizeModelConfig` and `ValidateModelConfig` - https://github.com/triton-inference-server/core/blob/r24.10/src/model_config_utils.cc#L734-L745 and https://github.com/triton-inference-server/core/blob/r24.10/src/model_config_utils.cc#L1410-L1498 - applies to r24.10. 6. Core dynamic batch scheduler (constructor at line 92; non-batching path at lines 240-251) - https://github.com/triton-inference-server/core/blob/r24.10/src/dynamic_batch_scheduler.cc - applies to r24.10. 7. Core dynamic batch scheduler on `main` (GitHub code search hit, commit 00e4e60) - https://github.com/triton-inference-server/core/blob/main/src/dynamic_batch_scheduler.cc - applies to main, 9 October 2026. 8. Python backend README (`execute`, Default Mode, Input Tensor Device Placement) - https://github.com/triton-inference-server/python_backend/blob/r24.10/README.md - applies to r24.10. 9. Python backend `CheckIncomingRequests` - https://github.com/triton-inference-server/python_backend/blob/r24.10/src/python_be.cc#L65-L113 - applies to r24.10.