Instructions to use SuperexponentialAI/relu with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Kernels
How to use SuperexponentialAI/relu with Kernels:
# !pip install kernels from kernels import get_kernel # a version (or an explicit revision) is required; see the "Files and versions" tab for the available ones kernel = get_kernel("SuperexponentialAI/relu", version=1) - Notebooks
- Google Colab
- Kaggle
v3: fix pre-Hopper (32MiB, L2] launch cliff (independently reported); retract position-biased 4090 claims; buffer-fair dtype benchmarks (MK, powered by Claude)
Browse files- L2_BAND_CLIFF.md +67 -0
- PERFORMANCE.md +16 -9
- README.md +54 -53
- benchmarks/benchmark.py +30 -13
- build.toml +1 -1
- build/torch212-cxx11-cu126-x86_64-linux/_ops.py +3 -3
- build/torch212-cxx11-cu126-x86_64-linux/{_relu_cuda_f03b26d.abi3.so → _relu_cuda_3c4de86.abi3.so} +2 -2
- build/torch212-cxx11-cu126-x86_64-linux/metadata.json +4 -4
- relu_cuda/relu.cu +30 -14
L2_BAND_CLIFF.md
ADDED
|
@@ -0,0 +1,67 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# The L2-band cliff: independently reported, reproduced, root-caused, fixed
|
| 2 |
+
|
| 3 |
+
**Status: fixed in v3** (`relu_cuda/relu.cu`, pre-Hopper band branch).
|
| 4 |
+
Credit: found by an independent recreation of the model-card results
|
| 5 |
+
(`../tests/RELU_KERNEL_RESULTS.md`) on RTX 4090 against the v2 release.
|
| 6 |
+
|
| 7 |
+
## The bug
|
| 8 |
+
|
| 9 |
+
v2's `launch_relu()` used evict-first streaming loads (`__ldcs`) for every
|
| 10 |
+
working set ≥ 32 MiB. On pre-Hopper GPUs, for working sets in
|
| 11 |
+
**(32 MiB, L2]** (4090: 72 MiB), that discards exactly the input-reuse
|
| 12 |
+
residency `torch.relu` exploits: torch serves repeated calls from L2 at up
|
| 13 |
+
to ~3.5 TB/s while the streaming kernel re-fetches from DRAM every call.
|
| 14 |
+
|
| 15 |
+
Measured impact (4090, correctness-gated, multiple methodologies —
|
| 16 |
+
independent report's single-workload sweep + our robust_bench pairwise):
|
| 17 |
+
**0.34–0.65x vs torch across the band, for every dtype at its own byte
|
| 18 |
+
sizes.** The band is byte-defined, so the same two nominal test sizes
|
| 19 |
+
(1024², 4096²) dodge it for fp32 (8 / 128 MiB) and land fp16/bf16's
|
| 20 |
+
"large" right inside it (64 MiB) — which is why the published v2 card
|
| 21 |
+
showed good fp32 numbers and the fp16/bf16 claims failed to reproduce.
|
| 22 |
+
|
| 23 |
+
## The fix (v3)
|
| 24 |
+
|
| 25 |
+
Working sets in (32 MiB, `l2CacheSize`] on `major < 9` now use **default
|
| 26 |
+
loads + streaming stores**: the re-read input stays L2-resident; the
|
| 27 |
+
write-once output still streams (evicting it early keeps more L2 for the
|
| 28 |
+
input). Above L2, streaming loads return — they genuinely win there
|
| 29 |
+
(1.13–1.30x vs torch at 76–84 MiB). Hopper is untouched: both of its band
|
| 30 |
+
points measured cliff-free with streaming (1.43–1.55x over torch; its
|
| 31 |
+
evict-first retention behaves differently).
|
| 32 |
+
|
| 33 |
+
Config selection data (4090, candidate-vs-torch anchored pairs, nsys):
|
| 34 |
+
|
| 35 |
+
| config @ band size | 33.6 MB | 52.4 MB | 75.5 MB |
|
| 36 |
+
|---|:---:|:---:|:---:|
|
| 37 |
+
| default ld + `__stcs` st (shipped) | 1.00x | **1.05x** | 1.05x |
|
| 38 |
+
| fully default | 1.01x | 0.99x | 1.07x |
|
| 39 |
+
| streaming (v2, control) | 0.40x | 0.54x | **1.13x** |
|
| 40 |
+
|
| 41 |
+
Post-fix validation: pairwise vs torch 0.99–1.06x across the band (was
|
| 42 |
+
0.40–0.54x); official harness fp16/bf16 `large` 1.02–1.03x (was 0.35x),
|
| 43 |
+
int8 `large` 1.14x (was 0.57x); no change outside the band on the 4090
|
| 44 |
+
(1024² 1.02x, ≥128 MiB 1.05x) and none on H100.
|
| 45 |
+
|
| 46 |
+
## Three measurement lessons this also exposed
|
| 47 |
+
|
| 48 |
+
1. **The v2 card's 4090 fp16/bf16 "1.85x/1.81x vs torch" were artifacts**
|
| 49 |
+
of a fixed-order shared-buffer interleave: ours always ran on input the
|
| 50 |
+
previous case had just re-cached, torch always ran on input our
|
| 51 |
+
evict-first pass had just evicted. Separate-buffer symmetric pairing
|
| 52 |
+
measures parity-to-1.3x depending on regime. Corrected everywhere.
|
| 53 |
+
2. **The allocating call pattern doubles the output footprint.**
|
| 54 |
+
`out = kernel.relu(x)` keeps the previous output alive at allocation
|
| 55 |
+
time, so the caching allocator alternates two output blocks; at
|
| 56 |
+
L2-boundary sizes that alone pushes a timed loop off the cliff.
|
| 57 |
+
`torch.relu` itself measures ~0.33x *against its own single-shot* under
|
| 58 |
+
that loop. The dtype benchmark class now uses a preallocated `out=`
|
| 59 |
+
and times the reference in the same form.
|
| 60 |
+
3. **Sweep coverage must be gap-free in bytes, not element counts.** The
|
| 61 |
+
original 4090 sweep jumped from 33.6 MB to 134 MB of working set —
|
| 62 |
+
the entire band containing the L2 boundary was never sampled, and the
|
| 63 |
+
32 MB threshold got extrapolated across it.
|
| 64 |
+
|
| 65 |
+
Repro scripts: `../tests/investigate_fp16_regression.py` (independent),
|
| 66 |
+
`bench/spec_pair_h100.py` + this session's candidate specs; raw data in
|
| 67 |
+
`../tests/relu_dead_zone_sweep.json`.
|
PERFORMANCE.md
CHANGED
|
@@ -7,17 +7,24 @@ methodology per platform: `OPTIMIZATION.md` (4090), `H100_BENCHMARK.md`
|
|
| 7 |
GPU-kernel durations, interleaved 1:1, correctness-gated (bit-identical
|
| 8 |
to `F.relu`; NaN → 0 matches upstream).
|
| 9 |
|
| 10 |
-
## CUDA — RTX 4090 (Ada, sm_89, tuned here)
|
| 11 |
|
| 12 |
-
|
| 13 |
-
|
| 14 |
-
|
| 15 |
-
|
| 16 |
-
|
| 17 |
-
| vs torch.relu int8 | — | ~parity (0.995x) |
|
| 18 |
|
| 19 |
-
|
| 20 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 21 |
|
| 22 |
## CUDA — H100 80GB HBM3 (sm_90, incl. same-day retune) [2026-07-30]
|
| 23 |
|
|
|
|
| 7 |
GPU-kernel durations, interleaved 1:1, correctness-gated (bit-identical
|
| 8 |
to `F.relu`; NaN → 0 matches upstream).
|
| 9 |
|
| 10 |
+
## CUDA — RTX 4090 (Ada, sm_89, tuned here) — corrected in v3
|
| 11 |
|
| 12 |
+
**v3 retraction + fix** (see `L2_BAND_CLIFF.md`): the previously published
|
| 13 |
+
fp16/bf16 "1.85x/1.81x" and fp32 "1.20x DRAM" vs torch were artifacts of a
|
| 14 |
+
fixed-order shared-buffer interleave, and v2 additionally had a real
|
| 15 |
+
launch-config cliff (0.34–0.65x vs torch) for working sets in
|
| 16 |
+
(32 MiB, 72 MiB L2]. v3 numbers, fair buffer handling:
|
|
|
|
| 17 |
|
| 18 |
+
| comparison | 1024² (L2) | 32 MiB–L2 band | DRAM-bound |
|
| 19 |
+
|---|:---:|:---:|:---:|
|
| 20 |
+
| vs torch.relu fp32 | ~parity (0.95–1.04x) | **0.99–1.06x** (was 0.34–0.65x) | ~parity–1.09x |
|
| 21 |
+
| vs torch fp16 / bf16 | **1.17–1.27x** | ~parity | ~parity |
|
| 22 |
+
| vs torch int8 | **1.35–1.37x** | **1.14x** | ~parity |
|
| 23 |
+
| vs upstream Hub (fp32) | 0.96–1.22x (run-sensitive) | — | parity (both at wall) |
|
| 24 |
+
|
| 25 |
+
DRAM wall ~944 GB/s (93.7% of 1008 peak), dtype-invariant — every config
|
| 26 |
+
hits it; upstream can't do fp16/bf16/int8 at all, which remains the main
|
| 27 |
+
4090 advantage over the Hub kernel.
|
| 28 |
|
| 29 |
## CUDA — H100 80GB HBM3 (sm_90, incl. same-day retune) [2026-07-30]
|
| 30 |
|
README.md
CHANGED
|
@@ -12,7 +12,7 @@ be used with the [`kernels` library](https://github.com/huggingface/kernels).
|
|
| 12 |
# make sure `kernels` is installed: `pip install -U kernels`
|
| 13 |
from kernels import get_kernel
|
| 14 |
|
| 15 |
-
kernel_module = get_kernel("SuperexponentialAI/relu", version=
|
| 16 |
relu = kernel_module.relu
|
| 17 |
|
| 18 |
relu(x)
|
|
@@ -40,18 +40,21 @@ bit-identical to `torch.nn.functional.relu` on non-NaN inputs.
|
|
| 40 |
|
| 41 |
### Results at a glance
|
| 42 |
|
| 43 |
-
GPU rows: nsys GPU-kernel duration
|
| 44 |
-
passes, correctness-gated (bit-identical to
|
| 45 |
-
(`kernels-community/relu`) is fp32-only, so other
|
| 46 |
-
baseline.
|
| 47 |
-
|
|
|
|
|
|
|
|
|
|
| 48 |
|
| 49 |
| target | dtype | vs torch | vs upstream Hub |
|
| 50 |
|---|---|---|---|
|
| 51 |
-
| RTX 4090 (sm_89) | fp32 | 1.
|
| 52 |
-
| RTX 4090 | fp16 / bf16 |
|
| 53 |
-
| RTX 4090 | int8 |
|
| 54 |
-
| H100 (sm_90) | fp32 | **1.
|
| 55 |
| H100 | fp16 / bf16 | **1.38x** L2, ~1.0–1.20x DRAM | — |
|
| 56 |
| H100 | int8 | **1.23–1.61x** | — |
|
| 57 |
| CPU (i9-13900K) | fp32/fp16/bf16/int8 | at DRAM bandwidth ceiling | — |
|
|
@@ -59,50 +62,47 @@ per-platform docs.
|
|
| 59 |
|
| 60 |
### CUDA (sm_70–sm_90, benchmarked on RTX 4090 / Ada and H100 / Hopper)
|
| 61 |
|
| 62 |
-
float4-vectorized,
|
| 63 |
-
|
| 64 |
-
|
| 65 |
-
|
| 66 |
-
|
| 67 |
-
|
| 68 |
-
|
| 69 |
-
|
| 70 |
-
|
| 71 |
-
|
| 72 |
-
|
| 73 |
-
|
| 74 |
-
|
| 75 |
-
|
| 76 |
-
|
| 77 |
-
|
| 78 |
-
|
| 79 |
-
|
| 80 |
-
|
| 81 |
-
|
| 82 |
-
|
| 83 |
-
|
| 84 |
-
|
| 85 |
-
|
| 86 |
-
|
| 87 |
-
see `INVESTIGATION_L2.md`): **fp32 1.20x faster, fp16 1.85x faster,
|
| 88 |
-
bf16 1.81x faster, int8 0.995x (near-parity)**. The int8 gap versus
|
| 89 |
-
torch's more-parallel launch heuristic (previously ~6.6%) was closed
|
| 90 |
-
to sub-1% by the `__vmaxs4` change.
|
| 91 |
-
- fp16/bf16/fp8/int8 throughput scales with the memory-bound
|
| 92 |
1/sizeof(dtype) rule: bf16/fp16 ≈ 2x fp32, int8/fp8 ≈ 4x fp32
|
| 93 |
-
intrinsically
|
| 94 |
-
(`DTYPE_THROUGHPUT.md`).
|
| 95 |
|
| 96 |
**H100 80GB HBM3 (sm_90)** — same method (nsys GPU-kernel duration,
|
| 97 |
interleaved 1:1, 3 independent passes, `H100_BENCHMARK.md`):
|
| 98 |
|
| 99 |
- DRAM-bound, this kernel runs at the HBM3 wall (~3.0 TB/s, ~90% of
|
| 100 |
peak). The upstream Hub kernel never reaches it (2.2–2.4 TB/s), so
|
| 101 |
-
the DRAM-bound margin is
|
| 102 |
-
upstream** (fp32 4096²/8192²)
|
| 103 |
-
- vs `torch`: fp32 **1.
|
| 104 |
-
DRAM-bound (both at the wall);
|
| 105 |
-
**1.
|
|
|
|
|
|
|
|
|
|
| 106 |
- Robustness: this kernel's launch is shape-agnostic. The upstream Hub
|
| 107 |
kernel launches `grid = numel/size(-1)` blocks, which collapses to a
|
| 108 |
single block on flat 1-D tensors — measured **140x slower** than this
|
|
@@ -147,18 +147,19 @@ upstream's fp32 base/large workloads (directly comparable), and
|
|
| 147 |
|
| 148 |
- Local checkout: `kernels benchmark ./relu-build` (all workloads verify
|
| 149 |
against `F.relu`).
|
| 150 |
-
- Remote (`kernels benchmark SuperexponentialAI/relu --version
|
| 151 |
requires this repo to be hosted as a `kernel`-type repo; it is
|
| 152 |
currently `model`-type (kernel-repo creation is restricted for this
|
| 153 |
account), so use the local form until it migrates. The same caveat
|
| 154 |
applies to the `get_kernel` snippet above — load via
|
| 155 |
`snapshot_download` + `sys.path` in the meantime.
|
| 156 |
|
| 157 |
-
|
| 158 |
-
|
| 159 |
-
|
| 160 |
-
|
| 161 |
-
|
|
|
|
| 162 |
|
| 163 |
Full methodology and results: see `OPTIMIZATION.md`, `DTYPE_THROUGHPUT.md`,
|
| 164 |
and `INVESTIGATION_L2.md` in this repository.
|
|
|
|
| 12 |
# make sure `kernels` is installed: `pip install -U kernels`
|
| 13 |
from kernels import get_kernel
|
| 14 |
|
| 15 |
+
kernel_module = get_kernel("SuperexponentialAI/relu", version=3)
|
| 16 |
relu = kernel_module.relu
|
| 17 |
|
| 18 |
relu(x)
|
|
|
|
| 40 |
|
| 41 |
### Results at a glance
|
| 42 |
|
| 43 |
+
GPU rows: nsys GPU-kernel duration or the official harness with matched
|
| 44 |
+
buffer handling, 3+ passes, correctness-gated (bit-identical to
|
| 45 |
+
`F.relu`). Upstream (`kernels-community/relu`) is fp32-only, so other
|
| 46 |
+
dtypes have no Hub baseline. **v3 corrects earlier 4090 claims** (fp16/
|
| 47 |
+
bf16 "1.85x/1.81x", fp32 "1.20x DRAM") that came from a position-biased
|
| 48 |
+
shared-buffer interleave, and fixes a real (32 MiB, L2] launch-config
|
| 49 |
+
cliff — full story in `L2_BAND_CLIFF.md`; methodology in
|
| 50 |
+
`PERFORMANCE.md`.
|
| 51 |
|
| 52 |
| target | dtype | vs torch | vs upstream Hub |
|
| 53 |
|---|---|---|---|
|
| 54 |
+
| RTX 4090 (sm_89) | fp32 | ~parity (0.95–1.09x) | 0.96–1.22x L2, parity DRAM |
|
| 55 |
+
| RTX 4090 | fp16 / bf16 | 1.17–1.27x base, ~parity large | — |
|
| 56 |
+
| RTX 4090 | int8 | **1.14–1.37x** | — |
|
| 57 |
+
| H100 (sm_90) | fp32 | **1.52x** L2, ~parity DRAM | **1.24–1.44x** DRAM |
|
| 58 |
| H100 | fp16 / bf16 | **1.38x** L2, ~1.0–1.20x DRAM | — |
|
| 59 |
| H100 | int8 | **1.23–1.61x** | — |
|
| 60 |
| CPU (i9-13900K) | fp32/fp16/bf16/int8 | at DRAM bandwidth ceiling | — |
|
|
|
|
| 62 |
|
| 63 |
### CUDA (sm_70–sm_90, benchmarked on RTX 4090 / Ada and H100 / Hopper)
|
| 64 |
|
| 65 |
+
float4-vectorized, flat grid-stride, scalar tail, arch-adaptive launch:
|
| 66 |
+
128-thread blocks below 32 MiB working sets, 512 above; streaming
|
| 67 |
+
`__ldcs`/`__stcs` **except** in the pre-Hopper (32 MiB, L2] band, which
|
| 68 |
+
uses default loads + streaming stores (v3 fix — evict-first loads there
|
| 69 |
+
ran 0.34–0.65x vs torch by discarding L2 input residency; see
|
| 70 |
+
`L2_BAND_CLIFF.md`), and Hopper+ below 16 MB, which uses unroll-2 /
|
| 71 |
+
stream-store-only / 256-thread blocks (+4.6–9.3%, symmetric pairwise
|
| 72 |
+
A/B). int8 additionally uses `__vmaxs4` SIMD byte-max. Tuned via
|
| 73 |
+
per-arch empirical sweeps, a 5-agent design fan-out, and roofline
|
| 74 |
+
analysis (`OPTIMIZATION.md` for sm_89, `H100_BENCHMARK.md` for sm_90 —
|
| 75 |
+
including which standalone-sweep "wins" failed in-harness verification
|
| 76 |
+
and were not shipped).
|
| 77 |
+
|
| 78 |
+
- **4090 vs `torch.relu`** (v3, fair buffer handling — see
|
| 79 |
+
`L2_BAND_CLIFF.md` for why v2's shared-buffer interleave numbers were
|
| 80 |
+
retracted): fp32 ~parity at every size (0.95–1.09x, both at the ~944
|
| 81 |
+
GB/s DRAM wall when DRAM-bound); fp16/bf16 1.17–1.27x at 1024²,
|
| 82 |
+
~parity at 4096²; int8 1.14–1.37x. In the band the v2 kernel lost
|
| 83 |
+
2–3x, v3 measures 0.99–1.06x.
|
| 84 |
+
- **4090 vs upstream `kernels-community/relu`** (fp32, bit-identical
|
| 85 |
+
output): 0.96–1.22x at L2-resident sizes (run-sensitive), parity at
|
| 86 |
+
the DRAM wall — both kernels saturate GDDR6X there. Upstream's real
|
| 87 |
+
gaps are dtype coverage (fp32-only) and the 1-D launch collapse
|
| 88 |
+
below.
|
| 89 |
+
- fp16/bf16/fp8/int8 element throughput scales with the memory-bound
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 90 |
1/sizeof(dtype) rule: bf16/fp16 ≈ 2x fp32, int8/fp8 ≈ 4x fp32
|
| 91 |
+
intrinsically (`DTYPE_THROUGHPUT.md`).
|
|
|
|
| 92 |
|
| 93 |
**H100 80GB HBM3 (sm_90)** — same method (nsys GPU-kernel duration,
|
| 94 |
interleaved 1:1, 3 independent passes, `H100_BENCHMARK.md`):
|
| 95 |
|
| 96 |
- DRAM-bound, this kernel runs at the HBM3 wall (~3.0 TB/s, ~90% of
|
| 97 |
peak). The upstream Hub kernel never reaches it (2.2–2.4 TB/s), so
|
| 98 |
+
the DRAM-bound margin is real and position-insensitive: **1.24–1.44x
|
| 99 |
+
vs upstream** (fp32 4096²/8192²).
|
| 100 |
+
- vs `torch` (separate-buffer symmetric pairs): fp32 **1.52x** at
|
| 101 |
+
L2-resident 1024², ~parity DRAM-bound (both at the wall); fp16/bf16
|
| 102 |
+
**1.38x** at 1024², 1.0–1.20x at 4096²; int8 **1.23x** at 1024²,
|
| 103 |
+
**1.61x** at L2-resident 4096². The pre-Hopper L2-band cliff does not
|
| 104 |
+
occur on Hopper (measured 1.43–1.55x over torch at band sizes with
|
| 105 |
+
the streaming config; `L2_BAND_CLIFF.md`).
|
| 106 |
- Robustness: this kernel's launch is shape-agnostic. The upstream Hub
|
| 107 |
kernel launches `grid = numel/size(-1)` blocks, which collapses to a
|
| 108 |
single block on flat 1-D tensors — measured **140x slower** than this
|
|
|
|
| 147 |
|
| 148 |
- Local checkout: `kernels benchmark ./relu-build` (all workloads verify
|
| 149 |
against `F.relu`).
|
| 150 |
+
- Remote (`kernels benchmark SuperexponentialAI/relu --version 3`)
|
| 151 |
requires this repo to be hosted as a `kernel`-type repo; it is
|
| 152 |
currently `model`-type (kernel-repo creation is restricted for this
|
| 153 |
account), so use the local form until it migrates. The same caveat
|
| 154 |
applies to the `get_kernel` snippet above — load via
|
| 155 |
`snapshot_download` + `sys.path` in the meantime.
|
| 156 |
|
| 157 |
+
`ReluDtypeBenchmark` uses the kernel's preallocated `out=` form and times
|
| 158 |
+
the reference the same way — the allocating form alternates two live
|
| 159 |
+
output blocks and pushes L2-boundary workloads off the L2 cliff for
|
| 160 |
+
whichever side allocates (details in `L2_BAND_CLIFF.md`). For pure GPU
|
| 161 |
+
kernel time, the nsys numbers above remain the ground truth
|
| 162 |
+
(`INVESTIGATION_L2.md`).
|
| 163 |
|
| 164 |
Full methodology and results: see `OPTIMIZATION.md`, `DTYPE_THROUGHPUT.md`,
|
| 165 |
and `INVESTIGATION_L2.md` in this repository.
|
benchmarks/benchmark.py
CHANGED
|
@@ -33,13 +33,23 @@ class ReluBenchmark(Benchmark):
|
|
| 33 |
|
| 34 |
class ReluDtypeBenchmark(Benchmark):
|
| 35 |
"""Dtypes this build ships beyond upstream's fp32-only: fp16, bf16, int8.
|
| 36 |
-
base = 1024^2, large = 4096^2, matching ReluBenchmark's shapes.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 37 |
|
| 38 |
seed: int = 42
|
| 39 |
|
| 40 |
def _float_setup(self, side, dtype):
|
| 41 |
self.x = torch.randn(side, side, device=self.device, dtype=dtype)
|
| 42 |
self.out = torch.empty_like(self.x)
|
|
|
|
| 43 |
|
| 44 |
def setup_fp16_base(self):
|
| 45 |
self._float_setup(1024, torch.float16)
|
|
@@ -58,6 +68,7 @@ class ReluDtypeBenchmark(Benchmark):
|
|
| 58 |
-128, 128, (side, side), device=self.device, dtype=torch.int8
|
| 59 |
)
|
| 60 |
self.out = torch.empty_like(self.x)
|
|
|
|
| 61 |
|
| 62 |
def setup_int8_base(self):
|
| 63 |
self._int8_setup(1024)
|
|
@@ -66,37 +77,43 @@ class ReluDtypeBenchmark(Benchmark):
|
|
| 66 |
self._int8_setup(4096)
|
| 67 |
|
| 68 |
def benchmark_fp16_base(self):
|
| 69 |
-
self.
|
| 70 |
|
| 71 |
def benchmark_fp16_large(self):
|
| 72 |
-
self.
|
| 73 |
|
| 74 |
def benchmark_bf16_base(self):
|
| 75 |
-
self.
|
| 76 |
|
| 77 |
def benchmark_bf16_large(self):
|
| 78 |
-
self.
|
| 79 |
|
| 80 |
def benchmark_int8_base(self):
|
| 81 |
-
self.
|
| 82 |
|
| 83 |
def benchmark_int8_large(self):
|
| 84 |
-
self.
|
| 85 |
|
| 86 |
def verify_fp16_base(self) -> torch.Tensor:
|
| 87 |
-
|
|
|
|
| 88 |
|
| 89 |
def verify_fp16_large(self) -> torch.Tensor:
|
| 90 |
-
|
|
|
|
| 91 |
|
| 92 |
def verify_bf16_base(self) -> torch.Tensor:
|
| 93 |
-
|
|
|
|
| 94 |
|
| 95 |
def verify_bf16_large(self) -> torch.Tensor:
|
| 96 |
-
|
|
|
|
| 97 |
|
| 98 |
def verify_int8_base(self) -> torch.Tensor:
|
| 99 |
-
|
|
|
|
| 100 |
|
| 101 |
def verify_int8_large(self) -> torch.Tensor:
|
| 102 |
-
|
|
|
|
|
|
| 33 |
|
| 34 |
class ReluDtypeBenchmark(Benchmark):
|
| 35 |
"""Dtypes this build ships beyond upstream's fp32-only: fp16, bf16, int8.
|
| 36 |
+
base = 1024^2, large = 4096^2, matching ReluBenchmark's shapes.
|
| 37 |
+
|
| 38 |
+
Uses the kernel's out= form with a preallocated output: the allocating
|
| 39 |
+
form (`self.out = kernel.relu(x)`) keeps the previous output alive at
|
| 40 |
+
allocation time, so the caching allocator alternates two output blocks
|
| 41 |
+
and the effective footprint grows by a full buffer — at L2-boundary
|
| 42 |
+
working sets that alone pushes the timed loop off the L2 cliff while
|
| 43 |
+
the single-shot reference timing stays on it (torch.relu itself
|
| 44 |
+
measures ~0.33x vs its own single-shot under the same loop). The out=
|
| 45 |
+
form keeps one live output, matching the reference's footprint."""
|
| 46 |
|
| 47 |
seed: int = 42
|
| 48 |
|
| 49 |
def _float_setup(self, side, dtype):
|
| 50 |
self.x = torch.randn(side, side, device=self.device, dtype=dtype)
|
| 51 |
self.out = torch.empty_like(self.x)
|
| 52 |
+
self.ref_out = torch.empty_like(self.x)
|
| 53 |
|
| 54 |
def setup_fp16_base(self):
|
| 55 |
self._float_setup(1024, torch.float16)
|
|
|
|
| 68 |
-128, 128, (side, side), device=self.device, dtype=torch.int8
|
| 69 |
)
|
| 70 |
self.out = torch.empty_like(self.x)
|
| 71 |
+
self.ref_out = torch.empty_like(self.x)
|
| 72 |
|
| 73 |
def setup_int8_base(self):
|
| 74 |
self._int8_setup(1024)
|
|
|
|
| 77 |
self._int8_setup(4096)
|
| 78 |
|
| 79 |
def benchmark_fp16_base(self):
|
| 80 |
+
self.kernel.relu(self.x, out=self.out)
|
| 81 |
|
| 82 |
def benchmark_fp16_large(self):
|
| 83 |
+
self.kernel.relu(self.x, out=self.out)
|
| 84 |
|
| 85 |
def benchmark_bf16_base(self):
|
| 86 |
+
self.kernel.relu(self.x, out=self.out)
|
| 87 |
|
| 88 |
def benchmark_bf16_large(self):
|
| 89 |
+
self.kernel.relu(self.x, out=self.out)
|
| 90 |
|
| 91 |
def benchmark_int8_base(self):
|
| 92 |
+
self.kernel.relu(self.x, out=self.out)
|
| 93 |
|
| 94 |
def benchmark_int8_large(self):
|
| 95 |
+
self.kernel.relu(self.x, out=self.out)
|
| 96 |
|
| 97 |
def verify_fp16_base(self) -> torch.Tensor:
|
| 98 |
+
torch.clamp(self.x, min=0, out=self.ref_out)
|
| 99 |
+
return self.ref_out
|
| 100 |
|
| 101 |
def verify_fp16_large(self) -> torch.Tensor:
|
| 102 |
+
torch.clamp(self.x, min=0, out=self.ref_out)
|
| 103 |
+
return self.ref_out
|
| 104 |
|
| 105 |
def verify_bf16_base(self) -> torch.Tensor:
|
| 106 |
+
torch.clamp(self.x, min=0, out=self.ref_out)
|
| 107 |
+
return self.ref_out
|
| 108 |
|
| 109 |
def verify_bf16_large(self) -> torch.Tensor:
|
| 110 |
+
torch.clamp(self.x, min=0, out=self.ref_out)
|
| 111 |
+
return self.ref_out
|
| 112 |
|
| 113 |
def verify_int8_base(self) -> torch.Tensor:
|
| 114 |
+
torch.clamp(self.x, min=0, out=self.ref_out)
|
| 115 |
+
return self.ref_out
|
| 116 |
|
| 117 |
def verify_int8_large(self) -> torch.Tensor:
|
| 118 |
+
torch.clamp(self.x, min=0, out=self.ref_out)
|
| 119 |
+
return self.ref_out
|
build.toml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
| 1 |
[general]
|
| 2 |
name = "relu"
|
| 3 |
license = "Apache-2.0"
|
| 4 |
-
version =
|
| 5 |
backends = [
|
| 6 |
"cpu",
|
| 7 |
"cuda",
|
|
|
|
| 1 |
[general]
|
| 2 |
name = "relu"
|
| 3 |
license = "Apache-2.0"
|
| 4 |
+
version = 3
|
| 5 |
backends = [
|
| 6 |
"cpu",
|
| 7 |
"cuda",
|
build/torch212-cxx11-cu126-x86_64-linux/_ops.py
CHANGED
|
@@ -1,9 +1,9 @@
|
|
| 1 |
import torch
|
| 2 |
-
from . import
|
| 3 |
-
ops = torch.ops.
|
| 4 |
|
| 5 |
def add_op_namespace_prefix(op_name: str):
|
| 6 |
"""
|
| 7 |
Prefix op by namespace.
|
| 8 |
"""
|
| 9 |
-
return f"
|
|
|
|
| 1 |
import torch
|
| 2 |
+
from . import _relu_cuda_3c4de86
|
| 3 |
+
ops = torch.ops._relu_cuda_3c4de86
|
| 4 |
|
| 5 |
def add_op_namespace_prefix(op_name: str):
|
| 6 |
"""
|
| 7 |
Prefix op by namespace.
|
| 8 |
"""
|
| 9 |
+
return f"_relu_cuda_3c4de86::{op_name}"
|
build/torch212-cxx11-cu126-x86_64-linux/{_relu_cuda_f03b26d.abi3.so → _relu_cuda_3c4de86.abi3.so}
RENAMED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:9e051f8ac8a1378546c97f96e66429a16f476d1bdfeedb4d9a563a06528ce2d2
|
| 3 |
+
size 457720
|
build/torch212-cxx11-cu126-x86_64-linux/metadata.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
| 1 |
{
|
| 2 |
"name": "relu",
|
| 3 |
-
"id": "
|
| 4 |
-
"version":
|
| 5 |
"license": "Apache-2.0",
|
| 6 |
"python-depends": [],
|
| 7 |
"backend": {
|
|
@@ -21,8 +21,8 @@
|
|
| 21 |
"algorithm": "sha256",
|
| 22 |
"files": {
|
| 23 |
"__init__.py": "lzFVikVVZVGdTURXq9skL4x50/8QcWTs99kuLNnXtG8=",
|
| 24 |
-
"_ops.py": "
|
| 25 |
-
"
|
| 26 |
"relu/__init__.py": "DFYPlrhXwYjEqCl/8n0SmWGZV8NFml5DPhMjKfv98GY="
|
| 27 |
}
|
| 28 |
}
|
|
|
|
| 1 |
{
|
| 2 |
"name": "relu",
|
| 3 |
+
"id": "_relu_cuda_3c4de86",
|
| 4 |
+
"version": 3,
|
| 5 |
"license": "Apache-2.0",
|
| 6 |
"python-depends": [],
|
| 7 |
"backend": {
|
|
|
|
| 21 |
"algorithm": "sha256",
|
| 22 |
"files": {
|
| 23 |
"__init__.py": "lzFVikVVZVGdTURXq9skL4x50/8QcWTs99kuLNnXtG8=",
|
| 24 |
+
"_ops.py": "0wm/ZKMPsoPIX+o9zUpD81Z04dcjtHEbfq+UDPEdACs=",
|
| 25 |
+
"_relu_cuda_3c4de86.abi3.so": "ngUfisihN4VGyX+W5mQpoW9HbRvf7ttNmlY6BlKM4tI=",
|
| 26 |
"relu/__init__.py": "DFYPlrhXwYjEqCl/8n0SmWGZV8NFml5DPhMjKfv98GY="
|
| 27 |
}
|
| 28 |
}
|
relu_cuda/relu.cu
CHANGED
|
@@ -12,9 +12,10 @@
|
|
| 12 |
// ReLU is memory-bound, so the design is what the config sweeps proved optimal:
|
| 13 |
// * 128-bit (int4) vectorized loads/stores -> 16/sizeof(T) elements per thread.
|
| 14 |
// * Streaming cache hints (__ldcs / __stcs) for write-once/read-once data.
|
| 15 |
-
// * Arch-adaptive launch (see launch_relu): blk128/512 + ld+st.cs
|
| 16 |
-
//
|
| 17 |
-
//
|
|
|
|
| 18 |
// Scalar tail for numel not divisible by the vec width.
|
| 19 |
// Supported dtypes: float32, float16, bfloat16, int8. Element throughput scales
|
| 20 |
// ~1/sizeof: fp16/bf16 ~2x fp32, int8 ~4x fp32 (all at the same ~945 GB/s wall).
|
|
@@ -95,22 +96,35 @@ static void launch_relu(torch::Tensor &out, torch::Tensor const &input,
|
|
| 95 |
|
| 96 |
// Base config (swept on RTX 4090, bench/sweep.cu): 128-thread blocks when
|
| 97 |
// L2-resident, 512 when DRAM-bound (32 MB = measured crossover), streaming
|
| 98 |
-
// loads and stores
|
| 99 |
int threads = (working_set < (int64_t{32} << 20)) ? 128 : 512;
|
| 100 |
int unroll = 1;
|
| 101 |
bool ldcs = true;
|
| 102 |
|
| 103 |
-
// Hopper+ small-working-set override. Of the standalone H100 sweep
|
| 104 |
-
// (bench/sweep_h100.cu), only this part survived in-harness verification
|
| 105 |
-
// (nsys interleaved, robust_bench): stream-store-only unroll-2 block-256 is
|
| 106 |
-
// +4.6% at 8.4 MB, but -27% at 33.6 MB under realistic L2 contention (the
|
| 107 |
-
// standalone sweep's +16-20% there did not reproduce, and neither did its
|
| 108 |
-
// DRAM-side blk1024 gain: 0 to -0.8% in-harness). 16 MB is a conservative
|
| 109 |
-
// cut between the verified 8.4 MB win and the verified 33.6 MB loss.
|
| 110 |
// Cached device props from torch -- no per-call driver query.
|
| 111 |
-
|
| 112 |
-
|
| 113 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 114 |
}
|
| 115 |
|
| 116 |
if (nvec > 0) {
|
|
@@ -121,6 +135,8 @@ static void launch_relu(torch::Tensor &out, torch::Tensor const &input,
|
|
| 121 |
auto const *i = reinterpret_cast<int4 const *>(ip);
|
| 122 |
if (unroll == 2 && !ldcs)
|
| 123 |
relu_vec<T, 2, false, true><<<blocks, threads, 0, stream>>>(o, i, nvec);
|
|
|
|
|
|
|
| 124 |
else
|
| 125 |
relu_vec<T, 1, true, true><<<blocks, threads, 0, stream>>>(o, i, nvec);
|
| 126 |
}
|
|
|
|
| 12 |
// ReLU is memory-bound, so the design is what the config sweeps proved optimal:
|
| 13 |
// * 128-bit (int4) vectorized loads/stores -> 16/sizeof(T) elements per thread.
|
| 14 |
// * Streaming cache hints (__ldcs / __stcs) for write-once/read-once data.
|
| 15 |
+
// * Arch-adaptive launch (see launch_relu): blk128/512 + ld+st.cs, except
|
| 16 |
+
// (a) pre-Hopper working sets in (32 MiB, L2], which use default loads +
|
| 17 |
+
// streaming stores (L2_BAND_CLIFF.md), and (b) Hopper+ below 16 MB, which
|
| 18 |
+
// uses stream-store-only unroll-2 blk256.
|
| 19 |
// Scalar tail for numel not divisible by the vec width.
|
| 20 |
// Supported dtypes: float32, float16, bfloat16, int8. Element throughput scales
|
| 21 |
// ~1/sizeof: fp16/bf16 ~2x fp32, int8 ~4x fp32 (all at the same ~945 GB/s wall).
|
|
|
|
| 96 |
|
| 97 |
// Base config (swept on RTX 4090, bench/sweep.cu): 128-thread blocks when
|
| 98 |
// L2-resident, 512 when DRAM-bound (32 MB = measured crossover), streaming
|
| 99 |
+
// loads and stores.
|
| 100 |
int threads = (working_set < (int64_t{32} << 20)) ? 128 : 512;
|
| 101 |
int unroll = 1;
|
| 102 |
bool ldcs = true;
|
| 103 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 104 |
// Cached device props from torch -- no per-call driver query.
|
| 105 |
+
const cudaDeviceProp *prop = at::cuda::getCurrentDeviceProperties();
|
| 106 |
+
if (prop->major >= 9) {
|
| 107 |
+
// Hopper+ small-working-set override. Of the standalone H100 sweep
|
| 108 |
+
// (bench/sweep_h100.cu), only this part survived in-harness verification:
|
| 109 |
+
// stream-store-only unroll-2 block-256, +4.6-9.3% below 16 MB. Hopper's
|
| 110 |
+
// evict-first retention is strong enough that the pre-Hopper L2-band
|
| 111 |
+
// cliff below does NOT occur there (measured: 1.43-1.55x over torch at
|
| 112 |
+
// 32-50 MiB working sets with the streaming config).
|
| 113 |
+
if (working_set < (int64_t{16} << 20)) {
|
| 114 |
+
threads = 256; unroll = 2; ldcs = false;
|
| 115 |
+
}
|
| 116 |
+
} else if (working_set >= (int64_t{32} << 20) &&
|
| 117 |
+
working_set <= static_cast<int64_t>(prop->l2CacheSize)) {
|
| 118 |
+
// Pre-Hopper L2-band fix (independently reported, reproduced, and
|
| 119 |
+
// root-caused 2026-07-31; see L2_BAND_CLIFF.md):
|
| 120 |
+
// with evict-first loads, working sets in (32 MiB, L2] run 2-3x SLOWER
|
| 121 |
+
// than torch.relu -- __ldcs discards exactly the input-reuse residency
|
| 122 |
+
// torch exploits (measured 0.34-0.65x vs torch across the band on the
|
| 123 |
+
// 4090's 72 MiB L2, for every dtype at its own byte sizes). Default
|
| 124 |
+
// loads restore parity-to-better; stores stay streaming (output is
|
| 125 |
+
// write-once -- evicting it early keeps more L2 for the input). Above
|
| 126 |
+
// L2, streaming loads win again (1.13-1.30x vs torch) and stay.
|
| 127 |
+
ldcs = false;
|
| 128 |
}
|
| 129 |
|
| 130 |
if (nvec > 0) {
|
|
|
|
| 135 |
auto const *i = reinterpret_cast<int4 const *>(ip);
|
| 136 |
if (unroll == 2 && !ldcs)
|
| 137 |
relu_vec<T, 2, false, true><<<blocks, threads, 0, stream>>>(o, i, nvec);
|
| 138 |
+
else if (!ldcs)
|
| 139 |
+
relu_vec<T, 1, false, true><<<blocks, threads, 0, stream>>>(o, i, nvec);
|
| 140 |
else
|
| 141 |
relu_vec<T, 1, true, true><<<blocks, threads, 0, stream>>>(o, i, nvec);
|
| 142 |
}
|