CLIP: Optimized for AMD ROCm

CLIP (Contrastive Language-Image Pre-training) performs zero-shot image classification by comparing image embeddings against text prompt embeddings. This repository packages CLIP evaluation on CIFAR-10 using PyTorch (CPU or GPU via OpenAI CLIP) and vLLM (GPU server with pooling runner, CPU HTTP client), exported and validated for AMD ROCm so it runs efficiently on AMD GPUs and CPUs.

This is based on the implementation of CLIP found here. This repository contains configurations and scripts optimized for AMD® ROCm™ platforms. You can use the AMD scripts to reproduce results or export with custom configurations. More details on model performance can be found here.

Task Overview

Task: Zero-shot image classification

Dataset: CIFAR-10 test split (10,000 images, 10 classes)

Output metrics: Zero-shot accuracy (%)

Quick Start: Before running this example, complete the main repository setup — see [Prerequisites](#prerequisites), [Configure System Paths](#2-configure-system-paths), and [Virtual Environments](#3-virtual-environments) in the main README.

Model variants: Default is base32 (openai/clip-vit-base-patch32 / ViT-B/32). Override with MODEL_VARIANT=base16|large14|large14-336.

vLLM note: vLLM's CLIP backend embeds one modality per request — text prompts and images are sent in separate API calls, then cosine similarity is computed client-side (same approach as the original evaluation scripts).

AMD ROCm Optimization

This model export has been adapted and validated for AMD Instinct™ / Radeon™ GPUs running ROCm, as well as AMD CPUs. Key points:

Exported/tested with ROCm <rocm-version> and PyTorch ROCm build <torch-rocm-version>.
Validated backends: PyTorch (native ROCm HIP kernels) and vLLM (ROCm-enabled server build).
No code changes required versus the upstream OpenAI CLIP implementation — only environment/runtime configuration differs.
CPU fallback path supported for environments without a ROCm-capable GPU.

Runtime	Precision	Backend	Hardware	Notes
PyTorch	fp32/fp16	HIP (ROCm)	AMD Instinct / Radeon GPU	Native OpenAI CLIP inference
PyTorch	fp32	CPU	AMD CPU (EPYC/Ryzen)	CPU-only fallback
vLLM	fp16	ROCm server	AMD Instinct GPU	Pooling runner, image-only requests
vLLM (client)	—	HTTP	AMD CPU	Text-prompt requests, CPU client

Getting Started

Option 1: Use Provided Scripts

Pre-configured evaluation scripts are available for direct use on ROCm hardware. See Quick Start above for setup steps.

Option 2: Run with Custom Configuration

Use the scripts in on GitHub to run with your own:

Custom model variant (base16, large14, large14-336, etc.)
Custom dataset (beyond CIFAR-10)
Target AMD GPU/CPU and runtime (PyTorch vs vLLM)

This option is ideal if you need to customize the evaluation beyond the default configuration provided here.

Model Details

Model Type: Zero-shot image classification (contrastive image-text embedding)

Base Model: openai/clip-vit-base-patch32 (ViT-B/32)

Model Stats:

Model variant: base32 (default) — base16 / large14 / large14-336 also supported
Image encoder: ViT-B/32
Text encoder: Transformer (CLIP text tower)
Input resolution: 224x224 (base variants), 336x336 (large14-336)
Number of parameters: <fill-in>
Precision tested: fp32, fp16

Performance Summary

Higher zero-shot accuracy means more test images are assigned the correct CIFAR-10 class via CLIP's image–text similarity — 100% is perfect, 10% is chance level for 10 classes. Values above ~85% on CIFAR-10 with ViT-B/32 are typical for this benchmark.

Metrics Explained

Metric	Description
Zero-shot accuracy (%)	Fraction of CIFAR-10 test images whose highest-scoring text prompt matches the ground-truth label after softmax over 10 class prompts. Primary accuracy metric; sensitive to both image and text embedding quality.

Accuracy Results

Full Dataset Evaluation (CIFAR-10 test) — filled from evaluation_results/; run make metrics to refresh:

Device	Backend	Precision	Variant	Accuracy (%)
CPU	PyTorch	FP16	clip-vit-base-patch32	88.79
CPU	PyTorch	FP32	clip-vit-base-patch32	88.80
GPU	PyTorch	FP16	clip-vit-base-patch32	88.75
GPU	PyTorch	FP32	clip-vit-base-patch32	88.80
GPU	vLLM	FP16	clip-vit-base-patch32	88.78
GPU	vLLM	FP32	clip-vit-base-patch32	88.80

Dig Deeper

Want to explore the full evaluation scripts, config options, and other AMD-optimized model examples?

📂 View the full project on GitHub

The GitHub repository includes:

Setup and prerequisites for ROCm environments
Scripts for both PyTorch and vLLM runners
Additional model variants and datasets
Benchmarking and reproduction instructions

Downloads last month: -; Downloads are not tracked for this model. How to track