---
title: GPU Inference Workbench
emoji: 🟩
colorFrom: green
colorTo: gray
sdk: static
app_file: index.html
license: apache-2.0
short_description: Triton config lab and LLM benchmark report viewer
tags:
- nvidia
- triton-inference-server
- tensorrt
- inference
- benchmark
- tutorial
- llm
- vllm
---

# GPU Inference Workbench

**Portfolio project by Kuo Jun Bin.**

A practical portfolio tool for NVIDIA inference workflows. Generate a starter NVIDIA Triton Inference Server `config.pbtxt`, or inspect an `inferbench` report and common vLLM-style LLM summary JSON.

This Space runs as a **free static website**. Both tools execute locally in your browser; there is no server, paid compute, GPU requirement, account token, or report upload.

## Start here

Use the three-step tour on the Space: build a Triton config, load the real batch-1 sample report, then follow the NVIDIA deployment runbook. The sample is from `results/bench-b1.json` in the source repository: ResNet-18 measured on an Apple M1 running macOS, with batch size 1, 100 iterations, and 20 warm-up iterations. CUDA, TensorRT, and Triton are explicitly unavailable on that host; these results are **not** NVIDIA GPU measurements.

## Tools

### Triton config builder

Generate a starter model configuration for ONNX Runtime or a TensorRT plan. Configure tensor names and dimensions, data type, maximum batch size, preferred batches, queue delay, instance count and CPU/GPU instance kind. Copy or download the generated `config.pbtxt`.

When `max_batch_size > 0`, input and output `dims` describe one sample and exclude the leading batch axis. Triton forms the full tensor shape by adding that axis. The builder disables dynamic batching when `max_batch_size` is zero.

### Benchmark report analyzer

Paste or load an `inferbench` report with a `results` array, a flat vLLM-style aggregate summary, or result rows containing recognized LLM metric fields. The analyzer displays TTFT, TPOT, ITL, request latency, request throughput, output-token throughput and token totals when present. Latency fields include common vLLM names such as `mean_ttft_ms`, `median_ttft_ms`, and `p99_ttft_ms`; nested metric objects with explicit units are also accepted. For non-LLM reports it retains the p50/p99 latency, inference throughput, backend availability and accuracy-drift view.

The parser rejects negative or non-finite measurements, invalid availability types, reversed p50/p99 values, and top-1 agreement outside 0–1. Reports are limited to 2 MiB and 1,000 result rows to protect browser responsiveness. Detailed per-request exports and every vendor-specific AIPerf/GenAI-Perf schema are not yet automatically imported; reduce them to an aggregate summary or normalize them to the documented fields first. The JSON is parsed locally in your browser and is not sent to a server.

Use **LOAD LLM SCHEMA DEMO** to see a synthetic format example. Its numbers are illustrative only, not benchmark results.

## NVIDIA workflow

1. Use the public source project to export a model and generate a Triton model repository:

   ```bash
   python -m venv .venv
   . .venv/bin/activate
   pip install -r requirements.txt
   PYTHONPATH=src python -m inferbench.cli export
   PYTHONPATH=src python -m inferbench.cli triton-repo --out model_repository
   ```

2. On a Linux host with a compatible NVIDIA driver and NVIDIA Container Toolkit, run Triton Server with GPU access:

   ```bash
   docker run --gpus=all --rm -p8000-8002:8000-8002 \
     -v "$PWD/model_repository:/models" \
     nvcr.io/nvidia/tritonserver:24.08-py3 \
     tritonserver --model-repository=/models
   ```

3. Install optional GPU dependencies and measure on the target machine. Preserve the generated JSON and load it into the Report Analyzer:

   ```bash
   pip install -r requirements-gpu.txt
   PYTHONPATH=src python -m inferbench.cli bench --batch 32 --iterations 200
   ```

   The CLI records the host and leaves unsupported backends unavailable instead of filling in guessed numbers. TensorRT and Triton results in the source repository are marked as not yet measured on NVIDIA hardware; this Space keeps that distinction explicit.

## Run locally

Serve the project folder with `python3 -m http.server 7860`, then open `http://localhost:7860`. No package installation or build step is required.

## Design notes

- Hugging Face Static Space (`sdk: static`); free and served without compute.
- Report processing happens in-browser only; no backend API or report persistence.
- Generated configuration is a starting point, not a guarantee that a model will load. Confirm model tensor names, shapes, backend support, and target hardware before deployment.
- Independent portfolio project; NVIDIA and Triton are trademarks of NVIDIA Corporation. This work is not affiliated with or endorsed by NVIDIA.

## References

- [Source benchmark and inference template](https://github.com/jjfishjj/tensorrt-triton-inference-template)
- [NVIDIA Triton model configuration](https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user_guide/model_configuration.html)
- [Hugging Face Static Spaces](https://huggingface.co/docs/hub/en/spaces-sdks-static)
- [vLLM bench serve CLI](https://docs.vllm.ai/en/stable/cli/bench/serve/)
- [NVIDIA AIPerf metrics reference](https://docs.nvidia.com/aiperf/reference/ai-perf-metrics-reference)
- [NVIDIA GenAI-Perf documentation](https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/perf_analyzer/genai-perf/README.html)

## License

Apache-2.0. See [LICENSE](LICENSE).