src/triton_inference/docs/reference/05-protocols-shared-memory.md

# Protocols, binary tensors, shared memory and message limits

This note covers how a client talks to Triton: HTTP/REST against gRPC under the KServe v2 protocol, the binary
tensor extension, compression, system and CUDA shared memory, and the size limits on each path, with a note on what
the Python backend does with the bytes once they arrive. The facts apply to Triton 24.10 (`r24.10` branches of
`server`, `client` and `python_backend`); later branches are named where behaviour changed. Checked 9 October 2026.

## HTTP/REST and gRPC

- Triton serves both protocols from the KServe v2 standard plus Triton's own extensions; gRPC adds a bidirectional
  streaming form of the inference call, and NVIDIA "typically recommend[s] using the unary version" unless a stream
  is needed (for example to keep a sequence on one server behind a load balancer) [1].
- Default ports are 8000 (HTTP), 8001 (gRPC) and 8002 (metrics) [2, 3, 4]. `--http-thread-count` sets the number of
  threads handling HTTP requests; the default is 8 [2, 3].
- I found no statement in the r24.10 protocol, optimisation or performance-tuning documents comparing HTTP with gRPC
  for a given request size, so there is no documented basis for expecting one to be faster at about 1.7 MB [1, 5].
- In the KServe gRPC protocol, sending tensors as `raw_input_contents` / `raw_output_contents` "will typically allow
  higher performance" than the typed `contents` fields, because of how protobuf allocation interacts with gRPC [6].

## Binary tensor data extension (HTTP)

- The extension lets an HTTP request or response carry tensors as raw bytes after the JSON header; Triton reports
  `binary_tensor_data` among its server extensions [7].
- Binary data is little-endian, row-major, with no stride or padding; each type uses its native size [7].
- `binary_data_size` (int64, bytes) on an input or output marks it as binary; `binary_data: true` on a requested
  output asks for binary; `binary_data_output: true` on the request asks for all outputs in binary unless overridden [7].
- The body is the JSON object followed by the tensors in the order they appear in the JSON. When any binary data is
  present, `Inference-Header-Content-Length` gives the JSON length and `Content-Length` the whole body [7].
- A "raw binary request" (`Inference-Header-Content-Length: 0`, no JSON) is allowed only for models with one input
  and at most one variable dimension [7].
- Without the extension, tensor data travels as a JSON array of numbers in row-major order [6]. Inference: for
  435,200 fp32 samples that means formatting and parsing every value as text, several times the 1.74 MB of raw bytes.

## Compression

- The r24.10 HTTP server decompresses request bodies sent with `Content-Encoding: gzip` or `deflate`, and compresses
  responses when the client sends `Accept-Encoding: gzip` or `deflate` [8].
- The client libraries expose request and response compression for HTTP and a compression algorithm for gRPC; the
  default is no compression in both [9]. On the server, gRPC response compression is set with
  `--grpc-infer-response-compression-level` [1].
- Inference: on one host, compressing fp32 audio only adds CPU time on both sides.

## Message size limits

- gRPC server: `MAX_GRPC_MESSAGE_SIZE = INT32_MAX` (2,147,483,647 bytes) in `src/common.h`, applied with
  `builder_.SetMaxMessageSize(MAX_GRPC_MESSAGE_SIZE)` [10, 11].
- Triton's own gRPC clients raise their limits to the same value: the Python client sets
  `grpc.max_send_message_length` and `grpc.max_receive_message_length` to `INT32_MAX`, and the C++ client calls
  `SetMaxSendMessageSize` and `SetMaxReceiveMessageSize` with it [12, 13]. A third-party gRPC client keeps its own
  library default unless it is raised (the value of that default is not covered by Triton's sources).
- HTTP in r24.10 has no option for a maximum request size, and I found no size check in its `http_server.cc` [2, 8].
  `--http-max-input-size` first appears on the r25.06 branch (absent on r24.11 to r25.05): "Maximum allowed HTTP
  request input size in bytes. For compressed requests, this also limits the decompressed size", default `1 << 26`
  = 67,108,864 bytes (64 MiB) [14, 15].
- Where it applies (source reading, not tested): in r25.06 the limit is checked for each input sent as JSON data and
  for a raw binary request; r26.05 adds checks on the JSON header length and on decompressed size. I found no check
  on tensor bytes sent with `binary_data_size` in either branch [21, 22].

## Shared memory extensions

- Instead of sending tensor bytes in the request, a client can name a registered region with the input or output
  parameters `shared_memory_region` and `shared_memory_byte_size` (both required) and `shared_memory_offset`
  (optional, default 0); the input's `data` field must then be absent [16]. System and CUDA region names share one
  namespace; there is no Windows support, and Jetson supports only system shared memory [16].
- System shared memory, HTTP endpoints [16]: `GET v2/systemsharedmemory[/region/${REGION_NAME}]/status`,
  `POST v2/systemsharedmemory/region/${REGION_NAME}/register` with body `{"key", "offset", "byte_size"}` (the key of
  the underlying memory object), and `POST v2/systemsharedmemory[/region/${REGION_NAME}]/unregister` (no name
  unregisters all regions).
- CUDA shared memory, HTTP endpoints [16]: the same three under `v2/cudasharedmemory`, with a register body
  `{"raw_handle": {"b64": ...}, "device_id", "byte_size"}`, where `raw_handle` is the serialised cudaIPC handle,
  base64-encoded, and `device_id` is the GPU on which it was created.
- The gRPC service has matching `SystemSharedMemory*` and `CudaSharedMemory*` register, status and unregister calls,
  and the same parameters go in `InferInputTensor` / `InferRequestedOutputTensor` [16].
- In the r24.10 server, a system region is opened with `shm_open(key)` and `mmap`, and a CUDA region with
  `cudaSetDevice(device_id)` and `cudaIpcOpenMemHandle` [17]. The client therefore creates the region first (POSIX
  shared memory, or device memory exported with `cudaIpcGetMemHandle`, which "gets an interprocess memory handle for
  an existing device memory allocation" [18]). Inference: both mechanisms are host-local, so the client must run on
  the same host and, in containers, share the IPC namespace or `/dev/shm` with Triton.
- The client README says shared memory "can significantly improve performance in some cases", and ships helper
  modules for Python plus C++ and Python examples [9].

## What the Python backend does with the bytes

- The Python backend talks to its stub process through shared memory: 1 MB per model instance by default, grown in
  1 MB chunks, set with `shm-default-byte-size` and `shm-growth-byte-size`; inside Docker, `--shm-size` must cover
  the inputs and outputs because Docker's default is 64 MB [19].
- By default every input is moved to CPU before the model sees it; `FORCE_CPU_ONLY_INPUT_TENSORS: "no"` lets inputs
  stay in GPU memory where they were last used [19].
- In `python_be.cc` r24.10, a CPU input is allocated in the stub's shared-memory pool and the request data is copied
  into it (`SaveToSharedMemory`, then `ProcessTensor` or `ReadInputTensor`) [20]. Inference from that code: data that
  arrives through client system shared memory is still copied once into the Python backend's own pool; it saves the
  HTTP transfer and parsing, not this copy. GPU inputs take a separate path through a CUDA memory pool [20].

## Unverified

- The default receive limit of a non-Triton gRPC client library (often quoted as 4 MiB) was not checked against a
  primary source.
- Whether HTTP keep-alive is used and how idle connections are handled by Triton's HTTP server is not documented in
  the r24.10 docs I read.
- No measurement of HTTP against gRPC for this request size was found; any speed difference is unknown.
- That `--http-max-input-size` does not limit binary-extension tensors is read from source; it was not tested.

## What this means for the diarization model

- Each block is two requests of about 1.7 MB each, because the engine computes masks between the calls and sends the
  same `SPAN` twice (repo facts). HTTP with binary tensors is the documented efficient HTTP form; JSON would be far
  slower to encode and parse (inference).
- gRPC has no documented speed advantage at this size. If the engine ever used gRPC, its client would need a
  receive and send limit above 1.7 MB per block, or k x 1.7 MB when blocks are packed (inference).
- On 24.10 there is no HTTP body limit. On 25.06 and 26.05 the 64 MiB default appears not to cover binary tensors,
  so it would not cap packed blocks sent the way the engine sends them; if a later release extends it, 64 MiB is
  about 38 blocks of 1.74 MB (67,108,864 / 1,740,800 = 38.5) [14, 21, 22].
- System shared memory would remove the HTTP body copy but not the Python backend's own copy, and it ties the
  gateway to the same host and IPC namespace as Triton (inference).
- CUDA shared memory would need the Rust engine, a CPU-side HTTP client today, to allocate device memory and export
  IPC handles itself (inference).
- The container's `--shm-size` must hold the Python backend pools: with 2 requests in flight per GPU, inputs are a
  few MB per model instance, far below Docker's 64 MB default, but packed requests grow this linearly (inference).

## Sources

1. Inference Protocols and APIs - https://github.com/triton-inference-server/server/blob/r24.10/docs/customization_guide/inference_protocols.md - r24.10.
2. `command_line_parser.cc` (option text) - https://github.com/triton-inference-server/server/blob/r24.10/src/command_line_parser.cc - r24.10.
3. `command_line_parser.h` (defaults: `http_port_{8000}`, `http_thread_cnt_{8}`, `metrics_port_{8002}`) -
   https://github.com/triton-inference-server/server/blob/r24.10/src/command_line_parser.h#L193-L216 - r24.10.
4. `grpc_server.h` (`port_{8001}`) - https://github.com/triton-inference-server/server/blob/r24.10/src/grpc/grpc_server.h#L52 - r24.10.
5. Optimization; Performance Tuning - https://github.com/triton-inference-server/server/blob/r24.10/docs/user_guide/optimization.md and
   https://github.com/triton-inference-server/server/blob/r24.10/docs/user_guide/performance_tuning.md - r24.10.
6. KServe Open Inference Protocol v2, required API (Tensor Data, gRPC raw contents) -
   https://github.com/kserve/kserve/blob/master/docs/predict-api/v2/required_api.md - KServe master, 9 October 2026.
7. Binary Tensor Data Extension - https://github.com/triton-inference-server/server/blob/r24.10/docs/protocol/extension_binary_data.md - r24.10.
8. `http_server.cc` (compression handling) - https://github.com/triton-inference-server/server/blob/r24.10/src/http_server.cc#L2545-L2577 - r24.10.
9. Triton client README (Compression, System/CUDA Shared Memory) - https://github.com/triton-inference-server/client/blob/r24.10/README.md - r24.10.
10. `common.h` - https://github.com/triton-inference-server/server/blob/r24.10/src/common.h#L48 - r24.10.
11. `grpc_server.cc` - https://github.com/triton-inference-server/server/blob/r24.10/src/grpc/grpc_server.cc#L2285 - r24.10.
12. Python gRPC client - https://github.com/triton-inference-server/client/blob/r24.10/src/python/library/tritonclient/grpc/_client.py#L51-L54 (and L201-L202) - r24.10.
13. C++ gRPC client - https://github.com/triton-inference-server/client/blob/r24.10/src/c%2B%2B/library/grpc_client.cc#L479-L480 - r24.10.
14. `command_line_parser.cc` (`--http-max-input-size`) - https://github.com/triton-inference-server/server/blob/r25.06/src/command_line_parser.cc#L502-L506 - r25.06 (absent on r24.10 to r25.05; present on main).
15. `common.h` (`HTTP_DEFAULT_MAX_INPUT_SIZE = 1 << 26`) - https://github.com/triton-inference-server/server/blob/r25.06/src/common.h#L58 - r25.06.
16. Shared-Memory Extension - https://github.com/triton-inference-server/server/blob/r24.10/docs/protocol/extension_shared_memory.md - r24.10.
17. `shared_memory_manager.cc` - https://github.com/triton-inference-server/server/blob/r24.10/src/shared_memory_manager.cc#L139-L256 - r24.10.
18. CUDA Runtime API, Device Management (`cudaIpcGetMemHandle`, `cudaIpcOpenMemHandle`) -
    https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__DEVICE.html - page shows CUDA 13.4.
19. Python backend README (Managing Shared Memory, Input Tensor Device Placement) -
    https://github.com/triton-inference-server/python_backend/blob/r24.10/README.md - r24.10.
20. Python backend `GetInputTensor` - https://github.com/triton-inference-server/python_backend/blob/r24.10/src/python_be.cc#L349-L556 - r24.10.
21. `http_server.cc` (uses of `max_input_size_` at L2812-L2822 and L3104-L3113) - https://github.com/triton-inference-server/server/blob/r25.06/src/http_server.cc - r25.06.
22. `http_server.cc` and `data_compressor.h` (JSON size, decompression limit) - https://github.com/triton-inference-server/server/blob/r26.05/src/http_server.cc and
    https://github.com/triton-inference-server/server/blob/r26.05/src/data_compressor.h - r26.05.