Spaces:
Paused
Paused
| # CUDA vs ROCm compatibility matrix for the retro inference layer | |
| Scope: running the vLLM server (`sensorflow/retro/inference/vllm_server/`) | |
| that backs the Retrospective Analyzer's `vllm` backend. | |
| **Honesty statement.** This matrix was written on a macOS (Apple Silicon) | |
| machine where **none of the GPU rows below can be executed or verified**. It | |
| is compiled from general knowledge of the CUDA/ROCm/vLLM ecosystems as of the | |
| author's knowledge cutoff. vLLM moves fast: **verify every row against the | |
| release notes of the vLLM version you actually install.** Rows marked | |
| UNCERTAIN are flagged rather than guessed. | |
| ## Platform support at a glance | |
| | Platform | vLLM | Notes | | |
| |---|---|---| | |
| | Linux + NVIDIA (CUDA 12.x) | Supported | Primary, best-tested target | | |
| | Linux + AMD (ROCm 6.x, MI200/MI300 class) | Supported | Official ROCm builds; narrower feature set | | |
| | Linux + AMD consumer (RDNA) | UNCERTAIN | Community/experimental; verify per release | | |
| | macOS (Apple Silicon or Intel) | **NOT SUPPORTED** | No CUDA, no ROCm; this dev machine. Use Ollama (Metal) locally | | |
| | Windows native | NOT SUPPORTED | WSL2 + CUDA is the workaround | | |
| | CPU-only Linux | Limited/experimental | A CPU backend exists but is not a serving-grade path; verify | | |
| ## Package installation | |
| | Aspect | CUDA (NVIDIA) | ROCm (AMD) | | |
| |---|---|---| | |
| | Install | `pip install vllm` (prebuilt wheels, CUDA 12.x) | No universal PyPI wheel historically; use AMD's ROCm docker images or build from source with ROCm 6.x. Verify current release — ROCm wheels have been appearing | | |
| | Driver floor | Driver >= 535-series for CUDA 12.x wheels (verify) | ROCm >= 6.0 for current vLLM; MI300 needs recent ROCm | | |
| | Attention kernels | FlashAttention-2 / FlashInfer prebuilt | ROCm FlashAttention port (Composable Kernel / Triton based); fewer head-dim/feature combos supported | | |
| ## PyTorch builds | |
| | Aspect | CUDA | ROCm | | |
| |---|---|---| | |
| | Wheel source | `pip install torch` (cu12x wheels on PyPI) | `pip install torch --index-url https://download.pytorch.org/whl/rocm6.x` | | |
| | Detection | `torch.version.cuda` set, `torch.cuda.is_available()` | `torch.version.hip` set; ROCm *masquerades as* `torch.cuda` (HIP) — `torch.cuda.is_available()` is True on ROCm builds | | |
| | Pitfall | CPU-only wheel silently installed if index not pinned | Mixing a CUDA wheel onto a ROCm host fails at runtime, not install time | | |
| ## Quantization support (verify against your vLLM release) | |
| | Method | CUDA | ROCm | | |
| |---|---|---| | |
| | AWQ | Supported (Ampere+; kernels tuned for 4-bit) | UNCERTAIN/limited — AWQ kernels historically CUDA-only; do not assume | | |
| | GPTQ | Supported (incl. Marlin kernels on newer GPUs) | Partial — basic GPTQ paths have worked via Triton; Marlin is CUDA-specific | | |
| | FP8 (W8A8) | Supported on Hopper/Ada (H100, L40S); emulated/absent on older | Supported on MI300 (native FP8); not on older CDNA | | |
| | INT8 (SmoothQuant-style) | Supported | UNCERTAIN — verify | | |
| | bitsandbytes | Supported (load-time) | NOT SUPPORTED historically | | |
| `compat.py` encodes a conservative subset of this table | |
| (`CUDA_QUANTIZATIONS` / `ROCM_QUANTIZATIONS`) and says "verify against the | |
| installed vLLM release" in its PASS messages. | |
| ## Memory management | |
| | Aspect | CUDA | ROCm | | |
| |---|---|---| | |
| | PagedAttention KV cache | Core feature, mature | Same architecture, HIP-compiled kernels | | |
| | `GPU_MEMORY_UTILIZATION` | Fraction of VRAM pre-allocated (default 0.9) | Same flag; MI300's 192 GB HBM changes sizing math | | |
| | Unified/managed memory | Not used by vLLM | Not used by vLLM | | |
| | Multi-GPU (TENSOR_PARALLEL_SIZE) | NCCL | RCCL (NCCL API-compatible); verify topology support (xGMI vs NVLink) | | |
| ## Kernel compatibility | |
| | Aspect | CUDA | ROCm | | |
| |---|---|---| | |
| | Core kernels | CUDA C++/CUTLASS + Triton | HIPified C++ + Composable Kernel + Triton-on-ROCm | | |
| | Triton | Mature | Works on ROCm but kernel autotuning/coverage lags; some fused ops fall back to slower paths | | |
| | CUDA Graphs | Used for decode speedups | HIP graph support has lagged; verify (`--enforce-eager` is the fallback) | | |
| | Custom all-reduce | CUDA-specific fast path | Typically disabled on ROCm; RCCL path used | | |
| ## What this repo can and cannot claim | |
| - CAN claim (verified here): vLLM is unsupported on this macOS machine — | |
| `env_detect.py` + `compat.py` prove it at runtime, and the test suite | |
| asserts it. | |
| - CANNOT claim: any vLLM latency/throughput number. `benchmark.py` produces | |
| them only when pointed at a real CUDA/ROCm host; no such run happened here. | |
| - Local real-inference numbers, if any appear in reports, come from an | |
| **Ollama server on Apple CPU/Metal** and are labeled as such. | |