README.md

---
library_name: kernels
license: apache-2.0
---

This is the repository card of `SuperexponentialAI/relu`. It was built to
be used with the [`kernels` library](https://github.com/huggingface/kernels).

## How to use

```python
# make sure `kernels` is installed: `pip install -U kernels`
from kernels import get_kernel

kernel_module = get_kernel("SuperexponentialAI/relu", version=3)
relu = kernel_module.relu

relu(x)
```

## Available functions

- `relu`

## Scope

This build covers **cpu, cuda, and xpu only**. `metal` and `rocm` are
declared in `kernels-community/relu` upstream but are intentionally
**not** included here — no Apple or AMD hardware was available to build
or validate them, and this repo only ships backends that were actually
optimized and benchmarked.

Each included backend supports **fp32, fp16, bf16, and int8** (upstream
`kernels-community/relu` is fp32-only). Semantics match upstream exactly,
including `NaN → 0` (not PyTorch's `NaN → NaN`) — this is a pre-existing
upstream behavior, intentionally preserved. All backends verified
bit-identical to `torch.nn.functional.relu` on non-NaN inputs.

## Optimizations

### Results at a glance

GPU rows: nsys GPU-kernel duration or the official harness with matched
buffer handling, 3+ passes, correctness-gated (bit-identical to
`F.relu`). Upstream (`kernels-community/relu`) is fp32-only, so other
dtypes have no Hub baseline. **v3 corrects earlier 4090 claims** (fp16/
bf16 "1.85x/1.81x", fp32 "1.20x DRAM") that came from a position-biased
shared-buffer interleave, and fixes a real (32 MiB, L2] launch-config
cliff — full story in `L2_BAND_CLIFF.md`; methodology in
`PERFORMANCE.md`.

| target | dtype | vs torch | vs upstream Hub |
|---|---|---|---|
| RTX 4090 (sm_89) | fp32 | ~parity (0.95–1.09x) | 0.96–1.22x L2, parity DRAM |
| RTX 4090 | fp16 / bf16 | 1.17–1.27x base, ~parity large | — |
| RTX 4090 | int8 | **1.14–1.37x** | — |
| H100 (sm_90) | fp32 | **1.52x** L2, ~parity DRAM | **1.24–1.44x** DRAM |
| H100 | fp16 / bf16 | **1.38x** L2, ~1.0–1.20x DRAM | — |
| H100 | int8 | **1.23–1.61x** | — |
| CPU (i9-13900K) | fp32/fp16/bf16/int8 | at DRAM bandwidth ceiling | — |
| XPU (UHD 770) | fp32/fp16/bf16/int8 | 1.01x / 0.96x / 1.16x / 1.06x | — |

### CUDA (sm_70–sm_90, benchmarked on RTX 4090 / Ada and H100 / Hopper)

float4-vectorized, flat grid-stride, scalar tail, arch-adaptive launch:
128-thread blocks below 32 MiB working sets, 512 above; streaming
`__ldcs`/`__stcs` **except** in the pre-Hopper (32 MiB, L2] band, which
uses default loads + streaming stores (v3 fix — evict-first loads there
ran 0.34–0.65x vs torch by discarding L2 input residency; see
`L2_BAND_CLIFF.md`), and Hopper+ below 16 MB, which uses unroll-2 /
stream-store-only / 256-thread blocks (+4.6–9.3%, symmetric pairwise
A/B). int8 additionally uses `__vmaxs4` SIMD byte-max. Tuned via
per-arch empirical sweeps, a 5-agent design fan-out, and roofline
analysis (`OPTIMIZATION.md` for sm_89, `H100_BENCHMARK.md` for sm_90 —
including which standalone-sweep "wins" failed in-harness verification
and were not shipped).

- **4090 vs `torch.relu`** (v3, fair buffer handling — see
  `L2_BAND_CLIFF.md` for why v2's shared-buffer interleave numbers were
  retracted): fp32 ~parity at every size (0.95–1.09x, both at the ~944
  GB/s DRAM wall when DRAM-bound); fp16/bf16 1.17–1.27x at 1024²,
  ~parity at 4096²; int8 1.14–1.37x. In the band the v2 kernel lost
  2–3x, v3 measures 0.99–1.06x.
- **4090 vs upstream `kernels-community/relu`** (fp32, bit-identical
  output): 0.96–1.22x at L2-resident sizes (run-sensitive), parity at
  the DRAM wall — both kernels saturate GDDR6X there. Upstream's real
  gaps are dtype coverage (fp32-only) and the 1-D launch collapse
  below.
- fp16/bf16/fp8/int8 element throughput scales with the memory-bound
  1/sizeof(dtype) rule: bf16/fp16 ≈ 2x fp32, int8/fp8 ≈ 4x fp32
  intrinsically (`DTYPE_THROUGHPUT.md`).

**H100 80GB HBM3 (sm_90)** — same method (nsys GPU-kernel duration,
interleaved 1:1, 3 independent passes, `H100_BENCHMARK.md`):

- DRAM-bound, this kernel runs at the HBM3 wall (~3.0 TB/s, ~90% of
  peak). The upstream Hub kernel never reaches it (2.2–2.4 TB/s), so
  the DRAM-bound margin is real and position-insensitive: **1.24–1.44x
  vs upstream** (fp32 4096²/8192²).
- vs `torch` (separate-buffer symmetric pairs): fp32 **1.52x** at
  L2-resident 1024², ~parity DRAM-bound (both at the wall); fp16/bf16
  **1.38x** at 1024², 1.0–1.20x at 4096²; int8 **1.23x** at 1024²,
  **1.61x** at L2-resident 4096². The pre-Hopper L2-band cliff does not
  occur on Hopper (measured 1.43–1.55x over torch at band sizes with
  the streaming config; `L2_BAND_CLIFF.md`).
- Robustness: this kernel's launch is shape-agnostic. The upstream Hub
  kernel launches `grid = numel/size(-1)` blocks, which collapses to a
  single block on flat 1-D tensors — measured **140x slower** than this
  kernel at 4096² elements on H100 (`relu(x.view(-1))`-shaped calls).

### CPU (benchmarked on i9-13900K)

Rewritten with AVX2, OpenMP, and non-temporal stores; extended to
fp16/bf16/int8. A thread/ILP sweep confirmed the implementation sits at
the memory-bandwidth ceiling rather than being compute- or
thread-count-limited.

### XPU (SYCL, validated on Intel UHD 770 iGPU)

Vectorized to 16 bytes/work-item for coalesced access, extended to
fp16/bf16/int8. Hits the DDR5 bandwidth wall on all dtypes tested;
int8 measured 2.6x the naive (non-vectorized) SYCL implementation.

**Live comparison vs `torch.relu`** (torch 2.12.0+xpu, Intel UHD 770 —
a consumer iGPU, not an officially-supported PyTorch XPU tier; both
implementations are bound by the same shared-DDR5 wall, so gains are
modest compared to CUDA/CPU):

| dtype | torch (GB/s) | ours (GB/s) | ratio (ours vs torch) |
|-------|:---:|:---:|:---:|
| fp32  | 40.6 | 40.8 | 1.01x (tied) |
| fp16  | 43.8 | 42.1 | 0.96x (torch ~4% faster) |
| bf16  | 36.6 | 42.5 | 1.16x faster |
| int8  | 78.0 | 82.9 | 1.06x faster |

All four dtypes verified bit-identical to `F.relu`. See `AITESTING.md`
for the environment setup and hardware-driver caveats needed to run
this (a separate `torch==2.12.0+xpu` venv, Level Zero runtime install,
device ACLs, and a CPU-generate-then-copy workaround for an on-device
fp16/bf16 creation bug on this GPU).

## Benchmarks

Official-harness scripts live in `benchmarks/`: `ReluBenchmark` mirrors
upstream's fp32 base/large workloads (directly comparable), and
`ReluDtypeBenchmark` adds the fp16/bf16/int8 workloads upstream lacks.

- Local checkout: `kernels benchmark ./relu-build` (all workloads verify
  against `F.relu`).
- Remote (`kernels benchmark SuperexponentialAI/relu --version 3`)
  requires this repo to be hosted as a `kernel`-type repo; it is
  currently `model`-type (kernel-repo creation is restricted for this
  account), so use the local form until it migrates. The same caveat
  applies to the `get_kernel` snippet above — load via
  `snapshot_download` + `sys.path` in the meantime.

`ReluDtypeBenchmark` uses the kernel's preallocated `out=` form and times
the reference the same way — the allocating form alternates two live
output blocks and pushes L2-boundary workloads off the L2 cliff for
whichever side allocates (details in `L2_BAND_CLIFF.md`). For pure GPU
kernel time, the nsys numbers above remain the ground truth
(`INVESTIGATION_L2.md`).

Full methodology and results: see `OPTIMIZATION.md`, `DTYPE_THROUGHPUT.md`,
and `INVESTIGATION_L2.md` in this repository.

## Upstream

The original source code for this kernel comes from
[kernels-community/relu](https://huggingface.co/kernels-community/relu).