| <!doctype html> |
| <html lang="en"> |
| <head> |
| <meta charset="utf-8"> |
| <meta name="viewport" content="width=device-width, initial-scale=1"> |
| <meta name="theme-color" content="#101411"> |
| <meta name="description" content="A hands-on inference lab: build Triton configs, explore real benchmark data, and learn the NVIDIA deployment path."> |
| <title>GPU Inference Workbench</title> |
| <link rel="stylesheet" href="./style.css"> |
| </head> |
| <body> |
| <header class="topbar"> |
| <a class="brand" href="#top" aria-label="GPU Inference Workbench home"><span class="brand-mark">IW</span><span>INFERENCE WORKBENCH</span></a> |
| <div class="topbar-right"><span class="status-dot"></span><span>REAL M1 DATA · NO GPU CLAIMS</span><a href="https://github.com/jjfishjj/tensorrt-triton-inference-template" target="_blank" rel="noreferrer">SOURCE ↗</a></div> |
| </header> |
| <main id="top"> |
| <section class="hero"> |
| <div class="eyebrow"><span>HANDS-ON INFERENCE LAB</span><span class="eyebrow-divider">/</span><span>TensorRT</span><span class="eyebrow-divider">/</span><span>Triton</span></div> |
| <h1>Benchmark honestly.<br><em>Deploy confidently.</em></h1> |
| <p class="hero-copy">A practical path from model shapes and batching to measured results and Triton serving. Explore real ResNet-18 results from an Apple M1, then follow the NVIDIA deployment runbook—without mistaking them for GPU measurements.</p> |
| <div class="hero-actions"> |
| <button class="button primary" id="start-tour" type="button">START THE 3-MIN TOUR <span>↘</span></button> |
| <button class="button" id="load-sample-hero" type="button">SEE REAL M1 RESULTS <span>↘</span></button> |
| <span class="hero-proof">OPEN SOURCE · APPLE M1 SAMPLE · NOT NVIDIA GPU RESULTS</span> |
| </div> |
| <div class="hero-notes" aria-label="Three-step learning path"> |
| <button class="path-card" data-open-tab="config" type="button"> |
| <span class="path-kicker"><b>01</b> CONFIGURE</span> |
| <strong>Shapes & batching</strong> |
| <span class="path-copy">Learn how Triton handles per-sample dimensions and dynamic batching.</span> |
| <span class="path-link">OPEN CONFIG BUILDER ↗</span> |
| </button> |
| <button class="path-card" data-open-tab="report" data-load-sample="true" type="button"> |
| <span class="path-kicker"><b>02</b> MEASURE</span> |
| <strong>Real results, real limits</strong> |
| <span class="path-copy">Explore p50/p99, throughput, accuracy drift, and unavailable GPU backends.</span> |
| <span class="path-link">LOAD M1 SAMPLE ↗</span> |
| </button> |
| <button class="path-card" data-open-tab="runbook" type="button"> |
| <span class="path-kicker"><b>03</b> DEPLOY</span> |
| <strong>Follow the NVIDIA path</strong> |
| <span class="path-copy">Connect model export, Triton serving, and repeatable measurement.</span> |
| <span class="path-link">OPEN RUNBOOK ↗</span> |
| </button> |
| </div> |
| </section> |
|
|
| <nav class="tabs" id="workbench" role="tablist" aria-label="Workbench tools"> |
| <button class="tab active" id="tab-config" role="tab" aria-selected="true" aria-controls="panel-config" data-tab="config" type="button"><span>01</span> CONFIG BUILDER</button> |
| <button class="tab" id="tab-report" role="tab" aria-selected="false" aria-controls="panel-report" data-tab="report" type="button"><span>02</span> REPORT ANALYZER</button> |
| <button class="tab" id="tab-runbook" role="tab" aria-selected="false" aria-controls="panel-runbook" data-tab="runbook" type="button"><span>03</span> NVIDIA RUNBOOK</button> |
| </nav> |
|
|
| <section class="tool-panel active" id="panel-config" role="tabpanel" aria-labelledby="tab-config" tabindex="0"> |
| <div class="panel-heading"><div><div class="section-label">TRITON INFERENCE SERVER · LESSON 01</div><h2>Model configuration builder</h2></div><p>Generate a starter <code>config.pbtxt</code>, then change batch settings to see how the model contract changes.</p></div> |
| <div class="config-layout"> |
| <form id="config-form" class="form-grid"> |
| <label class="field field-wide"><span>Model name</span><input name="model_name" value="resnet18" maxlength="128" required><small>Use a simple repository-safe name.</small></label> |
| <label class="field"><span>Model platform</span><select name="platform"><option value="onnxruntime_onnx">ONNX Runtime · ONNX</option><option value="tensorrt_plan">TensorRT · plan</option></select></label> |
| <label class="field"><span>Instance kind</span><select name="instance_kind"><option value="KIND_GPU">GPU</option><option value="KIND_CPU">CPU</option></select></label> |
| <label class="field"><span>Input tensor name</span><input name="input_name" value="input" required></label> |
| <label class="field"><span>Output tensor name</span><input name="output_name" value="output" required></label> |
| <label class="field"><span>Data type</span><select name="data_type"><option>TYPE_FP32</option><option>TYPE_FP16</option><option>TYPE_INT32</option><option>TYPE_INT64</option><option>TYPE_UINT8</option><option>TYPE_BOOL</option></select></label> |
| <label class="field"><span>Max batch size</span><input name="max_batch_size" type="number" value="32" min="0" max="4096" required><small>Try 0, then regenerate to compare.</small></label> |
| <label class="field"><span>Input dims</span><input name="input_dims" value="3, 224, 224" required><small>Per-sample dims; leave out the batch axis when max batch > 0.</small></label> |
| <label class="field"><span>Output dims</span><input name="output_dims" value="1000" required></label> |
| <label class="field"><span>Preferred batch sizes</span><input name="preferred_batch_sizes" value="4, 8, 16, 32"><small>Each value must be ≤ max batch size.</small></label> |
| <label class="field"><span>Queue delay (µs)</span><input name="queue_delay" type="number" value="100" min="0" max="100000"></label> |
| <label class="field"><span>Instance count</span><input name="instances" type="number" value="1" min="1" max="64"></label> |
| <div class="form-actions field-wide"><button class="button primary" type="submit">GENERATE CONFIG <span>↗</span></button><span class="form-hint">No model files are uploaded or stored.</span></div> |
| </form> |
| <div class="output-wrap"><div class="output-head"><div><span class="section-label">GENERATED CONFIG</span><span id="config-state" class="output-state">READY</span></div><div class="output-actions"><button id="copy-config" class="button small" type="button" disabled>COPY</button><button id="download-config" class="button small" type="button" disabled>DOWNLOAD</button></div></div><pre id="config-output" class="code-output"><code>Loading the worked example…</code></pre><div class="callout"><strong>WHY IT MATTERS</strong><span id="batch-explainer">Triton adds the leading batch dimension when <code>max_batch_size</code> is greater than zero. Input dimensions describe one sample.</span></div></div> |
| </div> |
| </section> |
|
|
| <section class="tool-panel" id="panel-report" role="tabpanel" aria-labelledby="tab-report" tabindex="0" hidden> |
| <div class="panel-heading"><div><div class="section-label">LESSON 02 · MEASURED RESULTS ONLY</div><h2>Benchmark report analyzer</h2></div><p>Load an inferbench report or a flat vLLM-style LLM summary. The Space parses results; it does not run a benchmark.</p></div> |
| <div class="sample-note"><span class="sample-tag">REAL SAMPLE</span><div><strong>ResNet-18 · Apple M1 · macOS · batch 1 · 100 iterations</strong><p>CUDA, TensorRT, and Triton were unavailable on the recorded host. These are not NVIDIA GPU results.</p></div><a href="https://github.com/jjfishjj/tensorrt-triton-inference-template/blob/main/results/bench-b1.json" target="_blank" rel="noreferrer">VIEW SOURCE ↗</a></div> |
| <div class="sample-note llm-sample-note"><span class="sample-tag">SCHEMA DEMO</span><div><strong>LLM metrics example · synthetic, not measured</strong><p>Illustrates TTFT, TPOT, ITL, request latency and token throughput fields. Use the button below to load this format example.</p></div></div> |
| <div class="report-inputs"><div class="input-actions"><button id="load-sample-report" class="button small" type="button">LOAD REAL M1 SAMPLE</button><button id="load-llm-sample" class="button small" type="button">LOAD LLM SCHEMA DEMO</button><label class="button small upload-button">LOAD JSON FILE<input id="report-file" type="file" accept=".json,application/json" hidden></label><button id="analyze-report" class="button primary" type="button">ANALYZE REPORT <span>↗</span></button><button id="clear-report" class="button small" type="button">CLEAR</button></div><textarea id="report-json" spellcheck="false" placeholder='Paste inferbench JSON or a flat vLLM-style summary with fields such as "mean_ttft_ms" and "output_token_throughput".'></textarea><div id="report-error" class="error-message" role="alert"></div></div> |
| <div id="report-results" class="report-results hidden" aria-live="polite"></div> |
| <div class="callout warning"><strong>READ THE EVIDENCE</strong><span>Unavailable backends are not zero-speed results. Read every number with its host, batch size, iteration count, and accuracy drift.</span></div> |
| </section> |
|
|
| <section class="tool-panel" id="panel-runbook" role="tabpanel" aria-labelledby="tab-runbook" tabindex="0" hidden> |
| <div class="panel-heading"><div><div class="section-label">LESSON 03 · FROM CONFIG TO SERVING</div><h2>NVIDIA deployment runbook</h2></div><p>A compact path from an ONNX model to Triton. Validate every engine on the target GPU before publishing performance claims.</p></div> |
| <div class="runbook-grid"> |
| <article class="runbook-card"><span class="step-number">01</span><h3>Generate & export</h3><p>Export a deterministic model to ONNX, then create a model repository with a matching version directory and configuration.</p><pre><code>python -m inferbench.cli export |
| python -m inferbench.cli triton-repo --out model_repository</code></pre></article> |
| <article class="runbook-card"><span class="step-number">02</span><h3>Serve with Triton</h3><p>Run the NVIDIA Triton Server container with GPU access and mount the generated model repository.</p><pre><code>docker run --gpus=all --rm \ |
| -p8000-8002:8000-8002 \ |
| -v "$PWD/model_repository:/models" \ |
| nvcr.io/nvidia/tritonserver:24.08-py3 \ |
| tritonserver --model-repository=/models</code></pre></article> |
| <article class="runbook-card"><span class="step-number">03</span><h3>Measure & compare</h3><p>Record the host, batch size, warm-up, iterations, latency percentiles, throughput, and accuracy drift for each available backend.</p><pre><code>python -m inferbench.cli bench \ |
| --batch 32 --iterations 200</code></pre></article> |
| </div> |
| <div class="callout"><strong>HARDWARE NOTE</strong><span>This Space runs in the visitor’s browser and uses no Hugging Face GPU. Use an NVIDIA GPU host for TensorRT and Triton measurements, then inspect the resulting JSON here.</span></div> |
| <p class="source-links"><a href="https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user_guide/model_configuration.html" target="_blank" rel="noreferrer">Triton model configuration docs ↗</a><a href="https://huggingface.co/docs/hub/en/spaces-sdks-docker" target="_blank" rel="noreferrer">Hugging Face Docker Spaces ↗</a><a href="https://github.com/jjfishjj/tensorrt-triton-inference-template" target="_blank" rel="noreferrer">Source repository ↗</a></p> |
| </section> |
|
|
| <footer class="footer"><div><span class="footer-mark">IW</span><span>GPU INFERENCE WORKBENCH</span></div><p>Built by Kuo Jun Bin · Adapted from the <a href="https://github.com/jjfishjj/tensorrt-triton-inference-template" target="_blank" rel="noreferrer">TensorRT / Triton inference template</a>.</p><p class="disclaimer">Independent portfolio project. NVIDIA and Triton are trademarks of NVIDIA Corporation. This project is not affiliated with or endorsed by NVIDIA.</p></footer> |
| </main> |
| <script type="module" src="./app.js"></script> |
| </body> |
| </html> |
|
|
