SuperexponentialAI/relu

This is the repository card of SuperexponentialAI/relu. It was built to be used with the kernels library.

How to use

# make sure `kernels` is installed: `pip install -U kernels`
from kernels import get_kernel

kernel_module = get_kernel("SuperexponentialAI/relu", version=3)
relu = kernel_module.relu

relu(x)

Available functions

• relu

Scope

This build covers cpu, cuda, and xpu only. metal and rocm are declared in kernels-community/relu upstream but are intentionally not included here — no Apple or AMD hardware was available to build or validate them, and this repo only ships backends that were actually optimized and benchmarked.

Each included backend supports fp32, fp16, bf16, and int8 (upstream kernels-community/relu is fp32-only). Semantics match upstream exactly, including NaN → 0 (not PyTorch's NaN → NaN) — this is a pre-existing upstream behavior, intentionally preserved. All backends verified bit-identical to torch.nn.functional.relu on non-NaN inputs.

Optimizations

Results at a glance

GPU rows: nsys GPU-kernel duration or the official harness with matched buffer handling, 3+ passes, correctness-gated (bit-identical to F.relu). Upstream (kernels-community/relu) is fp32-only, so other dtypes have no Hub baseline. v3 corrects earlier 4090 claims (fp16/ bf16 "1.85x/1.81x", fp32 "1.20x DRAM") that came from a position-biased shared-buffer interleave, and fixes a real (32 MiB, L2] launch-config cliff — full story in L2_BAND_CLIFF.md; methodology in PERFORMANCE.md.

target	dtype	vs torch	vs upstream Hub
RTX 4090 (sm_89)	fp32	~parity (0.95–1.09x)	0.96–1.22x L2, parity DRAM
RTX 4090	fp16 / bf16	1.17–1.27x base, ~parity large	—
RTX 4090	int8	1.14–1.37x	—
H100 (sm_90)	fp32	1.52x L2, ~parity DRAM	1.24–1.44x DRAM
H100	fp16 / bf16	1.38x L2, ~1.0–1.20x DRAM	—
H100	int8	1.23–1.61x	—
CPU (i9-13900K)	fp32/fp16/bf16/int8	at DRAM bandwidth ceiling	—
XPU (UHD 770)	fp32/fp16/bf16/int8	1.01x / 0.96x / 1.16x / 1.06x	—

CUDA (sm_70–sm_90, benchmarked on RTX 4090 / Ada and H100 / Hopper)

float4-vectorized, flat grid-stride, scalar tail, arch-adaptive launch: 128-thread blocks below 32 MiB working sets, 512 above; streaming __ldcs/__stcs except in the pre-Hopper (32 MiB, L2] band, which uses default loads + streaming stores (v3 fix — evict-first loads there ran 0.34–0.65x vs torch by discarding L2 input residency; see L2_BAND_CLIFF.md), and Hopper+ below 16 MB, which uses unroll-2 / stream-store-only / 256-thread blocks (+4.6–9.3%, symmetric pairwise A/B). int8 additionally uses __vmaxs4 SIMD byte-max. Tuned via per-arch empirical sweeps, a 5-agent design fan-out, and roofline analysis (OPTIMIZATION.md for sm_89, H100_BENCHMARK.md for sm_90 — including which standalone-sweep "wins" failed in-harness verification and were not shipped).

• 4090 vs torch.relu (v3, fair buffer handling — see L2_BAND_CLIFF.md for why v2's shared-buffer interleave numbers were retracted): fp32 ~parity at every size (0.95–1.09x, both at the ~944 GB/s DRAM wall when DRAM-bound); fp16/bf16 1.17–1.27x at 1024², ~parity at 4096²; int8 1.14–1.37x. In the band the v2 kernel lost 2–3x, v3 measures 0.99–1.06x.
• 4090 vs upstream kernels-community/relu (fp32, bit-identical output): 0.96–1.22x at L2-resident sizes (run-sensitive), parity at the DRAM wall — both kernels saturate GDDR6X there. Upstream's real gaps are dtype coverage (fp32-only) and the 1-D launch collapse below.
• fp16/bf16/fp8/int8 element throughput scales with the memory-bound 1/sizeof(dtype) rule: bf16/fp16 ≈ 2x fp32, int8/fp8 ≈ 4x fp32 intrinsically (DTYPE_THROUGHPUT.md).

H100 80GB HBM3 (sm_90) — same method (nsys GPU-kernel duration, interleaved 1:1, 3 independent passes, H100_BENCHMARK.md):

• DRAM-bound, this kernel runs at the HBM3 wall (~3.0 TB/s, ~90% of peak). The upstream Hub kernel never reaches it (2.2–2.4 TB/s), so the DRAM-bound margin is real and position-insensitive: 1.24–1.44x vs upstream (fp32 4096²/8192²).
• vs torch (separate-buffer symmetric pairs): fp32 1.52x at L2-resident 1024², ~parity DRAM-bound (both at the wall); fp16/bf16 1.38x at 1024², 1.0–1.20x at 4096²; int8 1.23x at 1024², 1.61x at L2-resident 4096². The pre-Hopper L2-band cliff does not occur on Hopper (measured 1.43–1.55x over torch at band sizes with the streaming config; L2_BAND_CLIFF.md).
• Robustness: this kernel's launch is shape-agnostic. The upstream Hub kernel launches grid = numel/size(-1) blocks, which collapses to a single block on flat 1-D tensors — measured 140x slower than this kernel at 4096² elements on H100 (relu(x.view(-1))-shaped calls).

CPU (benchmarked on i9-13900K)

Rewritten with AVX2, OpenMP, and non-temporal stores; extended to fp16/bf16/int8. A thread/ILP sweep confirmed the implementation sits at the memory-bandwidth ceiling rather than being compute- or thread-count-limited.

XPU (SYCL, validated on Intel UHD 770 iGPU)

Vectorized to 16 bytes/work-item for coalesced access, extended to fp16/bf16/int8. Hits the DDR5 bandwidth wall on all dtypes tested; int8 measured 2.6x the naive (non-vectorized) SYCL implementation.

Live comparison vs torch.relu (torch 2.12.0+xpu, Intel UHD 770 — a consumer iGPU, not an officially-supported PyTorch XPU tier; both implementations are bound by the same shared-DDR5 wall, so gains are modest compared to CUDA/CPU):

dtype	torch (GB/s)	ours (GB/s)	ratio (ours vs torch)
fp32	40.6	40.8	1.01x (tied)
fp16	43.8	42.1	0.96x (torch ~4% faster)
bf16	36.6	42.5	1.16x faster
int8	78.0	82.9	1.06x faster

All four dtypes verified bit-identical to F.relu. See AITESTING.md for the environment setup and hardware-driver caveats needed to run this (a separate torch==2.12.0+xpu venv, Level Zero runtime install, device ACLs, and a CPU-generate-then-copy workaround for an on-device fp16/bf16 creation bug on this GPU).

Benchmarks

Official-harness scripts live in benchmarks/: ReluBenchmark mirrors upstream's fp32 base/large workloads (directly comparable), and ReluDtypeBenchmark adds the fp16/bf16/int8 workloads upstream lacks.

• Local checkout: kernels benchmark ./relu-build (all workloads verify against F.relu).
• Remote (kernels benchmark SuperexponentialAI/relu --version 3) requires this repo to be hosted as a kernel-type repo; it is currently model-type (kernel-repo creation is restricted for this account), so use the local form until it migrates. The same caveat applies to the get_kernel snippet above — load via snapshot_download + sys.path in the meantime.

ReluDtypeBenchmark uses the kernel's preallocated out= form and times the reference the same way — the allocating form alternates two live output blocks and pushes L2-boundary workloads off the L2 cliff for whichever side allocates (details in L2_BAND_CLIFF.md). For pure GPU kernel time, the nsys numbers above remain the ground truth (INVESTIGATION_L2.md).

Full methodology and results: see OPTIMIZATION.md, DTYPE_THROUGHPUT.md, and INVESTIGATION_L2.md in this repository.

Upstream

The original source code for this kernel comes from kernels-community/relu.