Mapping the compose stack to Kubernetes
The compose stack (triton/compose/docker-compose.yml, scale.sh) runs one Triton per GPU, the gateway in front
deciding which GPU gets which block, and DCGM + Prometheus + Grafana. Two things shape the mapping: the gateway keeps
its jobs in memory, and its numerics depend on the binaries in its image. The same mapping applies to the serving
kit's FastAPI service (../../triton_inference/), which replaces the gateway.
| compose | Kubernetes | notes |
|---|---|---|
triton-i, NVIDIA_VISIBLE_DEVICES=<uuid>, cpuset |
a StatefulSet with resources.limits: nvidia.com/gpu: 1, replicas: N, CPU requests on the GPU's NUMA node (topologyManager: single-numa-node) |
one Pod = one GPU = one diar_nn_g<i> model at device 0; a StatefulSet gives each Pod the stable name the gateway's --triton list needs |
model repository per GPU (make_repository.py --gpus i --device 0) |
one ConfigMap or init container per ordinal writing diar_nn_g<ordinal> + resample_cpu |
the bundle (weights) as a read-only volume |
health check /v2/health/ready, 300 s start period |
startupProbe covering the self-test (every batch shape against the release, about a minute per GPU), readinessProbe /v2/health/ready, livenessProbe /v2/health/live |
a model whose self-test fails never loads, so the Pod never becomes ready |
stop_grace_period: 45s, --exit-timeout-secs=30 |
terminationGracePeriodSeconds: 45; Triton drains in-flight requests on SIGTERM |
plus a PodDisruptionBudget; the gateway resends any request cut by a restart (--triton-retry-s) |
the gateway on the host, --triton url0,url1,... (one per GPU) |
one Deployment with replicas: 1 on a CPU node, --triton listing the StatefulSet Pods' DNS names (headless Service) |
jobs live in the gateway's memory: several replicas need sticky routing by job id, or each replica gets its own Triton Pods. Numerics: the gateway image must carry the same glibc, numpy OpenBLAS and libsndfile binaries as the release's environment (the service image does), and the identity check (triton/identity_check.sh) or the kit's diar gate should pass in it before use |
/ready (503 while any model is away) |
the gateway's readinessProbe |
the gateway itself waits for Triton; clients back off on 429/503 |
| spool directory (0700) | an emptyDir (or local SSD) volume, --max-spool-gb below its size |
uploads never sit in memory |
--token / ANUVANI_DIAR_TOKEN |
a Secret as an environment variable; TLS at the Ingress |
never expose the gateway without a token |
| dcgm-exporter | the NVIDIA GPU Operator's dcgm-exporter DaemonSet | same counters (dcgm_counters.csv) |
| Prometheus, Grafana | kube-prometheus-stack, ServiceMonitors for each Triton Pod and the gateway; import grafana/dashboards/anuvani_diar.json |
|
scale.sh N |
kubectl scale statefulset --replicas=N and the gateway's --triton list (--gpus auto finds the models) |
the gateway needs a restart to see a new GPU; jobs survive it through the client's resubmission |
Sizing: each GPU's Triton model needs about 12 GB of GPU memory; throughput grows with the number of GPUs, and the
gateway's post-processing uses every CPU core it gets, so give it a node with many cores rather than pinning it to a
few. Measure a deployment with the kit's load test (../../triton_inference/tools/bench.py).