ResNet-50 (ONNX) — inference-optimization benchmark

This is a standard torchvision ResNet-50 (IMAGENET1K_V2 weights) exported to ONNX, published as the portable artifact from an inference-optimization study. The point of the repo is not the weights — they're the stock torchvision model — but the benchmark: what FP16, ONNX Runtime, and TensorRT (FP16 / INT8) do to latency and accuracy on the same hardware, measured the same way.

The .onnx file lets you reproduce the ONNX Runtime and TensorRT results, or build your own engine, without re-exporting.

Benchmark summary

NVIDIA GPU, CUDA 12.1, TensorRT 10.13. 200 timed iterations, 20 warm-up discarded, torch.cuda.synchronize() around each timed region. Accuracy on a fixed 3,200-image ImageNet-val slice, identical across variants.

Batch-1 latency (single-request):

Variant	p50 (ms)	Speedup vs FP32	Top-1
PyTorch FP32	6.87	1.0x	85.25%
PyTorch FP16	7.92	0.87x	85.25%
ONNX Runtime	2.67	2.6x	—
TensorRT FP16	0.74	9.3x	85.25%
TensorRT INT8	0.59	11.6x	84.97%

Batch-32 throughput:

Variant	Throughput (img/s)	vs FP32
PyTorch FP32	1068	1.0x
PyTorch FP16	2091	2.0x
ONNX Runtime	975	0.9x
TensorRT FP16	5118	4.8x
TensorRT INT8	10021	9.4x

Headline: TensorRT INT8 is 11.6x faster than FP32 at batch 1 for a 0.28-point top-1 drop.

The accuracy figures are higher than the canonical ~80.3% because they're on an easier 3,200-image slice; the relative gap between variants is the point, and it's valid because every variant saw the same images.

Intended use

Reference artifact for inference-optimization work: a fixed ResNet-50 ONNX graph you can run in ONNX Runtime or compile with TensorRT to reproduce or extend the numbers above. Not a new or improved model — the weights are stock torchvision.

How to use

ONNX Runtime (CUDA):

import onnxruntime as ort
import numpy as np

sess = ort.InferenceSession("resnet50.onnx", providers=["CUDAExecutionProvider"])
x = np.random.randn(1, 3, 224, 224).astype(np.float32)  # NCHW, ImageNet-normalized
out = sess.run(None, {sess.get_inputs()[0].name: x})[0]
pred = out.argmax(1)

Preprocessing is standard ImageNet eval: resize 256, center-crop 224, normalize with mean [0.485, 0.456, 0.406] and std [0.229, 0.224, 0.225], in NCHW order.

The input has a dynamic batch axis, so any batch size works.

Model details

Architecture: ResNet-50
Weights: torchvision ResNet50_Weights.IMAGENET1K_V2
Input: (batch, 3, 224, 224) float32, ImageNet-normalized, NCHW
Output: (batch, 1000) logits over ImageNet-1k classes
ONNX opset: 17

Limitations

Batch-1 p95 latency is noisy at sub-millisecond scale and shouldn't be read as a tail-latency guarantee. Absolute accuracy is on a subset, not the full 50k val set. INT8 numbers come from TensorRT 10.x implicit calibration (deprecated in 11.x, which uses explicit QDQ via NVIDIA ModelOpt).

Full code, benchmark scripts, and methodology: github.com/paulsaurav/resnet50-inference-optimization.

Downloads last month: -; Downloads are not tracked for this model. How to track

paulsaurav
/

resnet50-inference-optimization