README.md

metadata

title: GPU Inference Workbench
emoji: 🟩
colorFrom: green
colorTo: gray
sdk: static
app_file: index.html
license: apache-2.0
short_description: Triton config lab and LLM benchmark report viewer
tags:
  - nvidia
  - triton-inference-server
  - tensorrt
  - inference
  - benchmark
  - tutorial
  - llm
  - vllm

GPU Inference Workbench

Portfolio project by Kuo Jun Bin.

A practical portfolio tool for NVIDIA inference workflows. Generate a starter NVIDIA Triton Inference Server config.pbtxt, or inspect an inferbench report and common vLLM-style LLM summary JSON.

This Space runs as a free static website. Both tools execute locally in your browser; there is no server, paid compute, GPU requirement, account token, or report upload.

Start here

Use the three-step tour on the Space: build a Triton config, load the real batch-1 sample report, then follow the NVIDIA deployment runbook. The sample is from results/bench-b1.json in the source repository: ResNet-18 measured on an Apple M1 running macOS, with batch size 1, 100 iterations, and 20 warm-up iterations. CUDA, TensorRT, and Triton are explicitly unavailable on that host; these results are not NVIDIA GPU measurements.

Tools

Triton config builder

Generate a starter model configuration for ONNX Runtime or a TensorRT plan. Configure tensor names and dimensions, data type, maximum batch size, preferred batches, queue delay, instance count and CPU/GPU instance kind. Copy or download the generated config.pbtxt.

When max_batch_size > 0, input and output dims describe one sample and exclude the leading batch axis. Triton forms the full tensor shape by adding that axis. The builder disables dynamic batching when max_batch_size is zero.

Benchmark report analyzer

Paste or load an inferbench report with a results array, a flat vLLM-style aggregate summary, or result rows containing recognized LLM metric fields. The analyzer displays TTFT, TPOT, ITL, request latency, request throughput, output-token throughput and token totals when present. Latency fields include common vLLM names such as mean_ttft_ms, median_ttft_ms, and p99_ttft_ms; nested metric objects with explicit units are also accepted. For non-LLM reports it retains the p50/p99 latency, inference throughput, backend availability and accuracy-drift view.

The parser rejects negative or non-finite measurements, invalid availability types, reversed p50/p99 values, and top-1 agreement outside 0–1. Reports are limited to 2 MiB and 1,000 result rows to protect browser responsiveness. Detailed per-request exports and every vendor-specific AIPerf/GenAI-Perf schema are not yet automatically imported; reduce them to an aggregate summary or normalize them to the documented fields first. The JSON is parsed locally in your browser and is not sent to a server.

Use LOAD LLM SCHEMA DEMO to see a synthetic format example. Its numbers are illustrative only, not benchmark results.

NVIDIA workflow

1. Use the public source project to export a model and generate a Triton model repository:

   python -m venv .venv
   . .venv/bin/activate
   pip install -r requirements.txt
   PYTHONPATH=src python -m inferbench.cli export
   PYTHONPATH=src python -m inferbench.cli triton-repo --out model_repository
   

2. On a Linux host with a compatible NVIDIA driver and NVIDIA Container Toolkit, run Triton Server with GPU access:

   docker run --gpus=all --rm -p8000-8002:8000-8002 \
     -v "$PWD/model_repository:/models" \
     nvcr.io/nvidia/tritonserver:24.08-py3 \
     tritonserver --model-repository=/models
   

3. Install optional GPU dependencies and measure on the target machine. Preserve the generated JSON and load it into the Report Analyzer:

   pip install -r requirements-gpu.txt
   PYTHONPATH=src python -m inferbench.cli bench --batch 32 --iterations 200
   

   The CLI records the host and leaves unsupported backends unavailable instead of filling in guessed numbers. TensorRT and Triton results in the source repository are marked as not yet measured on NVIDIA hardware; this Space keeps that distinction explicit.

Run locally

Serve the project folder with python3 -m http.server 7860, then open http://localhost:7860. No package installation or build step is required.

Design notes

• Hugging Face Static Space (sdk: static); free and served without compute.
• Report processing happens in-browser only; no backend API or report persistence.
• Generated configuration is a starting point, not a guarantee that a model will load. Confirm model tensor names, shapes, backend support, and target hardware before deployment.
• Independent portfolio project; NVIDIA and Triton are trademarks of NVIDIA Corporation. This work is not affiliated with or endorsed by NVIDIA.

References

• Source benchmark and inference template
• NVIDIA Triton model configuration
• Hugging Face Static Spaces
• vLLM bench serve CLI
• NVIDIA AIPerf metrics reference
• NVIDIA GenAI-Perf documentation

License

Apache-2.0. See LICENSE.