CUDA vs ROCm compatibility matrix for the retro inference layer
Scope: running the vLLM server (sensorflow/retro/inference/vllm_server/)
that backs the Retrospective Analyzer's vllm backend.
Honesty statement. This matrix was written on a macOS (Apple Silicon)
machine where none of the GPU rows below can be executed or verified. It
is compiled from general knowledge of the CUDA/ROCm/vLLM ecosystems as of the
author's knowledge cutoff. vLLM moves fast: verify every row against the
release notes of the vLLM version you actually install. Rows marked
UNCERTAIN are flagged rather than guessed.
Platform support at a glance
| Platform |
vLLM |
Notes |
| Linux + NVIDIA (CUDA 12.x) |
Supported |
Primary, best-tested target |
| Linux + AMD (ROCm 6.x, MI200/MI300 class) |
Supported |
Official ROCm builds; narrower feature set |
| Linux + AMD consumer (RDNA) |
UNCERTAIN |
Community/experimental; verify per release |
| macOS (Apple Silicon or Intel) |
NOT SUPPORTED |
No CUDA, no ROCm; this dev machine. Use Ollama (Metal) locally |
| Windows native |
NOT SUPPORTED |
WSL2 + CUDA is the workaround |
| CPU-only Linux |
Limited/experimental |
A CPU backend exists but is not a serving-grade path; verify |
Package installation
| Aspect |
CUDA (NVIDIA) |
ROCm (AMD) |
| Install |
pip install vllm (prebuilt wheels, CUDA 12.x) |
No universal PyPI wheel historically; use AMD's ROCm docker images or build from source with ROCm 6.x. Verify current release — ROCm wheels have been appearing |
| Driver floor |
Driver >= 535-series for CUDA 12.x wheels (verify) |
ROCm >= 6.0 for current vLLM; MI300 needs recent ROCm |
| Attention kernels |
FlashAttention-2 / FlashInfer prebuilt |
ROCm FlashAttention port (Composable Kernel / Triton based); fewer head-dim/feature combos supported |
PyTorch builds
| Aspect |
CUDA |
ROCm |
| Wheel source |
pip install torch (cu12x wheels on PyPI) |
pip install torch --index-url https://download.pytorch.org/whl/rocm6.x |
| Detection |
torch.version.cuda set, torch.cuda.is_available() |
torch.version.hip set; ROCm masquerades as torch.cuda (HIP) — torch.cuda.is_available() is True on ROCm builds |
| Pitfall |
CPU-only wheel silently installed if index not pinned |
Mixing a CUDA wheel onto a ROCm host fails at runtime, not install time |
Quantization support (verify against your vLLM release)
| Method |
CUDA |
ROCm |
| AWQ |
Supported (Ampere+; kernels tuned for 4-bit) |
UNCERTAIN/limited — AWQ kernels historically CUDA-only; do not assume |
| GPTQ |
Supported (incl. Marlin kernels on newer GPUs) |
Partial — basic GPTQ paths have worked via Triton; Marlin is CUDA-specific |
| FP8 (W8A8) |
Supported on Hopper/Ada (H100, L40S); emulated/absent on older |
Supported on MI300 (native FP8); not on older CDNA |
| INT8 (SmoothQuant-style) |
Supported |
UNCERTAIN — verify |
| bitsandbytes |
Supported (load-time) |
NOT SUPPORTED historically |
compat.py encodes a conservative subset of this table
(CUDA_QUANTIZATIONS / ROCM_QUANTIZATIONS) and says "verify against the
installed vLLM release" in its PASS messages.
Memory management
| Aspect |
CUDA |
ROCm |
| PagedAttention KV cache |
Core feature, mature |
Same architecture, HIP-compiled kernels |
GPU_MEMORY_UTILIZATION |
Fraction of VRAM pre-allocated (default 0.9) |
Same flag; MI300's 192 GB HBM changes sizing math |
| Unified/managed memory |
Not used by vLLM |
Not used by vLLM |
| Multi-GPU (TENSOR_PARALLEL_SIZE) |
NCCL |
RCCL (NCCL API-compatible); verify topology support (xGMI vs NVLink) |
Kernel compatibility
| Aspect |
CUDA |
ROCm |
| Core kernels |
CUDA C++/CUTLASS + Triton |
HIPified C++ + Composable Kernel + Triton-on-ROCm |
| Triton |
Mature |
Works on ROCm but kernel autotuning/coverage lags; some fused ops fall back to slower paths |
| CUDA Graphs |
Used for decode speedups |
HIP graph support has lagged; verify (--enforce-eager is the fallback) |
| Custom all-reduce |
CUDA-specific fast path |
Typically disabled on ROCm; RCCL path used |
What this repo can and cannot claim
- CAN claim (verified here): vLLM is unsupported on this macOS machine —
env_detect.py + compat.py prove it at runtime, and the test suite
asserts it.
- CANNOT claim: any vLLM latency/throughput number.
benchmark.py produces
them only when pointed at a real CUDA/ROCm host; no such run happened here.
- Local real-inference numbers, if any appear in reports, come from an
Ollama server on Apple CPU/Metal and are labeled as such.
