Kernels
superexpai commited on
Commit
f30427a
·
verified ·
1 Parent(s): a7f6a13

v3: fix pre-Hopper (32MiB, L2] launch cliff (independently reported); retract position-biased 4090 claims; buffer-fair dtype benchmarks (MK, powered by Claude)

Browse files
L2_BAND_CLIFF.md ADDED
@@ -0,0 +1,67 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # The L2-band cliff: independently reported, reproduced, root-caused, fixed
2
+
3
+ **Status: fixed in v3** (`relu_cuda/relu.cu`, pre-Hopper band branch).
4
+ Credit: found by an independent recreation of the model-card results
5
+ (`../tests/RELU_KERNEL_RESULTS.md`) on RTX 4090 against the v2 release.
6
+
7
+ ## The bug
8
+
9
+ v2's `launch_relu()` used evict-first streaming loads (`__ldcs`) for every
10
+ working set ≥ 32 MiB. On pre-Hopper GPUs, for working sets in
11
+ **(32 MiB, L2]** (4090: 72 MiB), that discards exactly the input-reuse
12
+ residency `torch.relu` exploits: torch serves repeated calls from L2 at up
13
+ to ~3.5 TB/s while the streaming kernel re-fetches from DRAM every call.
14
+
15
+ Measured impact (4090, correctness-gated, multiple methodologies —
16
+ independent report's single-workload sweep + our robust_bench pairwise):
17
+ **0.34–0.65x vs torch across the band, for every dtype at its own byte
18
+ sizes.** The band is byte-defined, so the same two nominal test sizes
19
+ (1024², 4096²) dodge it for fp32 (8 / 128 MiB) and land fp16/bf16's
20
+ "large" right inside it (64 MiB) — which is why the published v2 card
21
+ showed good fp32 numbers and the fp16/bf16 claims failed to reproduce.
22
+
23
+ ## The fix (v3)
24
+
25
+ Working sets in (32 MiB, `l2CacheSize`] on `major < 9` now use **default
26
+ loads + streaming stores**: the re-read input stays L2-resident; the
27
+ write-once output still streams (evicting it early keeps more L2 for the
28
+ input). Above L2, streaming loads return — they genuinely win there
29
+ (1.13–1.30x vs torch at 76–84 MiB). Hopper is untouched: both of its band
30
+ points measured cliff-free with streaming (1.43–1.55x over torch; its
31
+ evict-first retention behaves differently).
32
+
33
+ Config selection data (4090, candidate-vs-torch anchored pairs, nsys):
34
+
35
+ | config @ band size | 33.6 MB | 52.4 MB | 75.5 MB |
36
+ |---|:---:|:---:|:---:|
37
+ | default ld + `__stcs` st (shipped) | 1.00x | **1.05x** | 1.05x |
38
+ | fully default | 1.01x | 0.99x | 1.07x |
39
+ | streaming (v2, control) | 0.40x | 0.54x | **1.13x** |
40
+
41
+ Post-fix validation: pairwise vs torch 0.99–1.06x across the band (was
42
+ 0.40–0.54x); official harness fp16/bf16 `large` 1.02–1.03x (was 0.35x),
43
+ int8 `large` 1.14x (was 0.57x); no change outside the band on the 4090
44
+ (1024² 1.02x, ≥128 MiB 1.05x) and none on H100.
45
+
46
+ ## Three measurement lessons this also exposed
47
+
48
+ 1. **The v2 card's 4090 fp16/bf16 "1.85x/1.81x vs torch" were artifacts**
49
+ of a fixed-order shared-buffer interleave: ours always ran on input the
50
+ previous case had just re-cached, torch always ran on input our
51
+ evict-first pass had just evicted. Separate-buffer symmetric pairing
52
+ measures parity-to-1.3x depending on regime. Corrected everywhere.
53
+ 2. **The allocating call pattern doubles the output footprint.**
54
+ `out = kernel.relu(x)` keeps the previous output alive at allocation
55
+ time, so the caching allocator alternates two output blocks; at
56
+ L2-boundary sizes that alone pushes a timed loop off the cliff.
57
+ `torch.relu` itself measures ~0.33x *against its own single-shot* under
58
+ that loop. The dtype benchmark class now uses a preallocated `out=`
59
+ and times the reference in the same form.
60
+ 3. **Sweep coverage must be gap-free in bytes, not element counts.** The
61
+ original 4090 sweep jumped from 33.6 MB to 134 MB of working set —
62
+ the entire band containing the L2 boundary was never sampled, and the
63
+ 32 MB threshold got extrapolated across it.
64
+
65
+ Repro scripts: `../tests/investigate_fp16_regression.py` (independent),
66
+ `bench/spec_pair_h100.py` + this session's candidate specs; raw data in
67
+ `../tests/relu_dead_zone_sweep.json`.
PERFORMANCE.md CHANGED
@@ -7,17 +7,24 @@ methodology per platform: `OPTIMIZATION.md` (4090), `H100_BENCHMARK.md`
7
  GPU-kernel durations, interleaved 1:1, correctness-gated (bit-identical
8
  to `F.relu`; NaN → 0 matches upstream).
9
 
10
- ## CUDA — RTX 4090 (Ada, sm_89, tuned here)
11
 
12
- | comparison | L2-resident | DRAM-bound |
13
- |---|:---:|:---:|
14
- | vs upstream Hub kernel (fp32) | **1.50x** | **1.16x** |
15
- | vs torch.relu fp32 | beats (2335 vs 2285 GB/s) | **1.20x** |
16
- | vs torch.relu fp16 / bf16 | — | **1.85x / 1.81x** |
17
- | vs torch.relu int8 | — | ~parity (0.995x) |
18
 
19
- DRAM wall ~944 GB/s (93.7% of 1008 peak) — every config hits it; the win
20
- is at L2-resident sizes (float4 + `__ldcs`/`__stcs` + adaptive block).
 
 
 
 
 
 
 
 
21
 
22
  ## CUDA — H100 80GB HBM3 (sm_90, incl. same-day retune) [2026-07-30]
23
 
 
7
  GPU-kernel durations, interleaved 1:1, correctness-gated (bit-identical
8
  to `F.relu`; NaN → 0 matches upstream).
9
 
10
+ ## CUDA — RTX 4090 (Ada, sm_89, tuned here) — corrected in v3
11
 
12
+ **v3 retraction + fix** (see `L2_BAND_CLIFF.md`): the previously published
13
+ fp16/bf16 "1.85x/1.81x" and fp32 "1.20x DRAM" vs torch were artifacts of a
14
+ fixed-order shared-buffer interleave, and v2 additionally had a real
15
+ launch-config cliff (0.34–0.65x vs torch) for working sets in
16
+ (32 MiB, 72 MiB L2]. v3 numbers, fair buffer handling:
 
17
 
18
+ | comparison | 1024² (L2) | 32 MiB–L2 band | DRAM-bound |
19
+ |---|:---:|:---:|:---:|
20
+ | vs torch.relu fp32 | ~parity (0.95–1.04x) | **0.99–1.06x** (was 0.34–0.65x) | ~parity–1.09x |
21
+ | vs torch fp16 / bf16 | **1.17–1.27x** | ~parity | ~parity |
22
+ | vs torch int8 | **1.35–1.37x** | **1.14x** | ~parity |
23
+ | vs upstream Hub (fp32) | 0.96–1.22x (run-sensitive) | — | parity (both at wall) |
24
+
25
+ DRAM wall ~944 GB/s (93.7% of 1008 peak), dtype-invariant — every config
26
+ hits it; upstream can't do fp16/bf16/int8 at all, which remains the main
27
+ 4090 advantage over the Hub kernel.
28
 
29
  ## CUDA — H100 80GB HBM3 (sm_90, incl. same-day retune) [2026-07-30]
30
 
README.md CHANGED
@@ -12,7 +12,7 @@ be used with the [`kernels` library](https://github.com/huggingface/kernels).
12
  # make sure `kernels` is installed: `pip install -U kernels`
13
  from kernels import get_kernel
14
 
15
- kernel_module = get_kernel("SuperexponentialAI/relu", version=2)
16
  relu = kernel_module.relu
17
 
18
  relu(x)
@@ -40,18 +40,21 @@ bit-identical to `torch.nn.functional.relu` on non-NaN inputs.
40
 
41
  ### Results at a glance
42
 
43
- GPU rows: nsys GPU-kernel duration, interleaved 1:1, 3 independent
44
- passes, correctness-gated (bit-identical to `F.relu`). Upstream
45
- (`kernels-community/relu`) is fp32-only, so other dtypes have no Hub
46
- baseline. Full methodology and caveats: `PERFORMANCE.md` and the
47
- per-platform docs.
 
 
 
48
 
49
  | target | dtype | vs torch | vs upstream Hub |
50
  |---|---|---|---|
51
- | RTX 4090 (sm_89) | fp32 | 1.20x DRAM, ties L2 | **1.50x** L2, 1.16x DRAM |
52
- | RTX 4090 | fp16 / bf16 | **1.85x / 1.81x** | — |
53
- | RTX 4090 | int8 | ~1.00x (parity) | — |
54
- | H100 (sm_90) | fp32 | **1.79x** L2, 1.01–1.10x DRAM | **1.53x** L2, 1.24–1.44x DRAM |
55
  | H100 | fp16 / bf16 | **1.38x** L2, ~1.0–1.20x DRAM | — |
56
  | H100 | int8 | **1.23–1.61x** | — |
57
  | CPU (i9-13900K) | fp32/fp16/bf16/int8 | at DRAM bandwidth ceiling | — |
@@ -59,50 +62,47 @@ per-platform docs.
59
 
60
  ### CUDA (sm_70–sm_90, benchmarked on RTX 4090 / Ada and H100 / Hopper)
61
 
62
- float4-vectorized, streaming `__ldcs`/`__stcs` load/store, flat
63
- grid-stride, scalar tail, arch-adaptive launch: 128-thread blocks for
64
- L2-resident sizes and 512 for DRAM-bound everywhere, except Hopper+
65
- below a 16 MB working set, which uses unroll-2 / stream-store-only /
66
- 256-thread blocks (+9.3% there, verified by symmetric pairwise A/B).
67
- int8 additionally uses `__vmaxs4` SIMD byte-max. Tuned via per-arch
68
- empirical sweeps, a 5-agent design fan-out, and roofline analysis
69
- (`OPTIMIZATION.md` for sm_89, `H100_BENCHMARK.md` for sm_90 — including
70
- which standalone-sweep "wins" failed in-harness verification and were
71
- not shipped).
72
-
73
- - **GPU-kernel-duration comparison vs upstream `kernels-community/relu`
74
- v1** (nsys, interleaved 1:1, fp32, bit-identical output): **1.50x
75
- faster at L2-resident sizes (1024², ~2700 vs ~1800 GB/s), 1.16x
76
- faster DRAM-bound (4096², ~1084 vs ~935 GB/s)**. This is a larger
77
- margin than earlier host-timed numbers suggested, because host
78
- wall-clock/`kernels benchmark` timing bakes in Python dispatch
79
- overhead that's roughly equal for both kernels and compresses the
80
- visible gap — nsys measures pure GPU kernel time. Absolute GB/s at
81
- the large size exceeds the 1008 GB/s theoretical DRAM ceiling because
82
- buffers are reused across iterations (evict-first `__ldcs`/`__stcs`
83
- partially keeps input L2-resident); treat the **ratio**, not the raw
84
- GB/s, as the trustworthy number there.
85
- - **GPU-kernel-duration comparison vs `torch.relu`** (nsys, interleaved
86
- 1:1, 16.7M elements — wall-clock timing is unreliable at this scale,
87
- see `INVESTIGATION_L2.md`): **fp32 1.20x faster, fp16 1.85x faster,
88
- bf16 1.81x faster, int8 0.995x (near-parity)**. The int8 gap versus
89
- torch's more-parallel launch heuristic (previously ~6.6%) was closed
90
- to sub-1% by the `__vmaxs4` change.
91
- - fp16/bf16/fp8/int8 throughput scales with the memory-bound
92
  1/sizeof(dtype) rule: bf16/fp16 ≈ 2x fp32, int8/fp8 ≈ 4x fp32
93
- intrinsically, up to ~8–15x at L2-resident working-set sizes
94
- (`DTYPE_THROUGHPUT.md`).
95
 
96
  **H100 80GB HBM3 (sm_90)** — same method (nsys GPU-kernel duration,
97
  interleaved 1:1, 3 independent passes, `H100_BENCHMARK.md`):
98
 
99
  - DRAM-bound, this kernel runs at the HBM3 wall (~3.0 TB/s, ~90% of
100
  peak). The upstream Hub kernel never reaches it (2.2–2.4 TB/s), so
101
- the DRAM-bound margin is *larger* than on the 4090: **1.24–1.44x vs
102
- upstream** (fp32 4096²/8192²), 1.53x at L2-resident 1024².
103
- - vs `torch`: fp32 **1.79x** at L2-resident 1024², 1.01–1.10x
104
- DRAM-bound (both at the wall); at 1024², fp16/bf16 **1.38x**, int8
105
- **1.23x**; int8 4096² (L2-resident) **1.61x**.
 
 
 
106
  - Robustness: this kernel's launch is shape-agnostic. The upstream Hub
107
  kernel launches `grid = numel/size(-1)` blocks, which collapses to a
108
  single block on flat 1-D tensors — measured **140x slower** than this
@@ -147,18 +147,19 @@ upstream's fp32 base/large workloads (directly comparable), and
147
 
148
  - Local checkout: `kernels benchmark ./relu-build` (all workloads verify
149
  against `F.relu`).
150
- - Remote (`kernels benchmark SuperexponentialAI/relu --version 2`)
151
  requires this repo to be hosted as a `kernel`-type repo; it is
152
  currently `model`-type (kernel-repo creation is restricted for this
153
  account), so use the local form until it migrates. The same caveat
154
  applies to the `get_kernel` snippet above — load via
155
  `snapshot_download` + `sys.path` in the meantime.
156
 
157
- The harness times host-side wall clock per iteration; on boost-clocked
158
- GPUs at L2-resident sizes this understates the kernel's true speedup
159
- relative to the single-shot reference timing. The nsys
160
- GPU-kernel-duration numbers above are the trustworthy comparison
161
- (`INVESTIGATION_L2.md` explains the mechanism).
 
162
 
163
  Full methodology and results: see `OPTIMIZATION.md`, `DTYPE_THROUGHPUT.md`,
164
  and `INVESTIGATION_L2.md` in this repository.
 
12
  # make sure `kernels` is installed: `pip install -U kernels`
13
  from kernels import get_kernel
14
 
15
+ kernel_module = get_kernel("SuperexponentialAI/relu", version=3)
16
  relu = kernel_module.relu
17
 
18
  relu(x)
 
40
 
41
  ### Results at a glance
42
 
43
+ GPU rows: nsys GPU-kernel duration or the official harness with matched
44
+ buffer handling, 3+ passes, correctness-gated (bit-identical to
45
+ `F.relu`). Upstream (`kernels-community/relu`) is fp32-only, so other
46
+ dtypes have no Hub baseline. **v3 corrects earlier 4090 claims** (fp16/
47
+ bf16 "1.85x/1.81x", fp32 "1.20x DRAM") that came from a position-biased
48
+ shared-buffer interleave, and fixes a real (32 MiB, L2] launch-config
49
+ cliff — full story in `L2_BAND_CLIFF.md`; methodology in
50
+ `PERFORMANCE.md`.
51
 
52
  | target | dtype | vs torch | vs upstream Hub |
53
  |---|---|---|---|
54
+ | RTX 4090 (sm_89) | fp32 | ~parity (0.95–1.09x) | 0.96–1.22x L2, parity DRAM |
55
+ | RTX 4090 | fp16 / bf16 | 1.17–1.27x base, ~parity large | — |
56
+ | RTX 4090 | int8 | **1.14–1.37x** | — |
57
+ | H100 (sm_90) | fp32 | **1.52x** L2, ~parity DRAM | **1.24–1.44x** DRAM |
58
  | H100 | fp16 / bf16 | **1.38x** L2, ~1.0–1.20x DRAM | — |
59
  | H100 | int8 | **1.23–1.61x** | — |
60
  | CPU (i9-13900K) | fp32/fp16/bf16/int8 | at DRAM bandwidth ceiling | — |
 
62
 
63
  ### CUDA (sm_70–sm_90, benchmarked on RTX 4090 / Ada and H100 / Hopper)
64
 
65
+ float4-vectorized, flat grid-stride, scalar tail, arch-adaptive launch:
66
+ 128-thread blocks below 32 MiB working sets, 512 above; streaming
67
+ `__ldcs`/`__stcs` **except** in the pre-Hopper (32 MiB, L2] band, which
68
+ uses default loads + streaming stores (v3 fix — evict-first loads there
69
+ ran 0.34–0.65x vs torch by discarding L2 input residency; see
70
+ `L2_BAND_CLIFF.md`), and Hopper+ below 16 MB, which uses unroll-2 /
71
+ stream-store-only / 256-thread blocks (+4.6–9.3%, symmetric pairwise
72
+ A/B). int8 additionally uses `__vmaxs4` SIMD byte-max. Tuned via
73
+ per-arch empirical sweeps, a 5-agent design fan-out, and roofline
74
+ analysis (`OPTIMIZATION.md` for sm_89, `H100_BENCHMARK.md` for sm_90 —
75
+ including which standalone-sweep "wins" failed in-harness verification
76
+ and were not shipped).
77
+
78
+ - **4090 vs `torch.relu`** (v3, fair buffer handling — see
79
+ `L2_BAND_CLIFF.md` for why v2's shared-buffer interleave numbers were
80
+ retracted): fp32 ~parity at every size (0.95–1.09x, both at the ~944
81
+ GB/s DRAM wall when DRAM-bound); fp16/bf16 1.17–1.27x at 1024²,
82
+ ~parity at 4096²; int8 1.14–1.37x. In the band the v2 kernel lost
83
+ 2–3x, v3 measures 0.99–1.06x.
84
+ - **4090 vs upstream `kernels-community/relu`** (fp32, bit-identical
85
+ output): 0.96–1.22x at L2-resident sizes (run-sensitive), parity at
86
+ the DRAM wall — both kernels saturate GDDR6X there. Upstream's real
87
+ gaps are dtype coverage (fp32-only) and the 1-D launch collapse
88
+ below.
89
+ - fp16/bf16/fp8/int8 element throughput scales with the memory-bound
 
 
 
 
 
90
  1/sizeof(dtype) rule: bf16/fp16 ≈ 2x fp32, int8/fp8 ≈ 4x fp32
91
+ intrinsically (`DTYPE_THROUGHPUT.md`).
 
92
 
93
  **H100 80GB HBM3 (sm_90)** — same method (nsys GPU-kernel duration,
94
  interleaved 1:1, 3 independent passes, `H100_BENCHMARK.md`):
95
 
96
  - DRAM-bound, this kernel runs at the HBM3 wall (~3.0 TB/s, ~90% of
97
  peak). The upstream Hub kernel never reaches it (2.2–2.4 TB/s), so
98
+ the DRAM-bound margin is real and position-insensitive: **1.24–1.44x
99
+ vs upstream** (fp32 4096²/8192²).
100
+ - vs `torch` (separate-buffer symmetric pairs): fp32 **1.52x** at
101
+ L2-resident 1024², ~parity DRAM-bound (both at the wall); fp16/bf16
102
+ **1.38x** at 1024², 1.0–1.20x at 4096²; int8 **1.23x** at 1024²,
103
+ **1.61x** at L2-resident 4096². The pre-Hopper L2-band cliff does not
104
+ occur on Hopper (measured 1.43–1.55x over torch at band sizes with
105
+ the streaming config; `L2_BAND_CLIFF.md`).
106
  - Robustness: this kernel's launch is shape-agnostic. The upstream Hub
107
  kernel launches `grid = numel/size(-1)` blocks, which collapses to a
108
  single block on flat 1-D tensors — measured **140x slower** than this
 
147
 
148
  - Local checkout: `kernels benchmark ./relu-build` (all workloads verify
149
  against `F.relu`).
150
+ - Remote (`kernels benchmark SuperexponentialAI/relu --version 3`)
151
  requires this repo to be hosted as a `kernel`-type repo; it is
152
  currently `model`-type (kernel-repo creation is restricted for this
153
  account), so use the local form until it migrates. The same caveat
154
  applies to the `get_kernel` snippet above — load via
155
  `snapshot_download` + `sys.path` in the meantime.
156
 
157
+ `ReluDtypeBenchmark` uses the kernel's preallocated `out=` form and times
158
+ the reference the same way — the allocating form alternates two live
159
+ output blocks and pushes L2-boundary workloads off the L2 cliff for
160
+ whichever side allocates (details in `L2_BAND_CLIFF.md`). For pure GPU
161
+ kernel time, the nsys numbers above remain the ground truth
162
+ (`INVESTIGATION_L2.md`).
163
 
164
  Full methodology and results: see `OPTIMIZATION.md`, `DTYPE_THROUGHPUT.md`,
165
  and `INVESTIGATION_L2.md` in this repository.
benchmarks/benchmark.py CHANGED
@@ -33,13 +33,23 @@ class ReluBenchmark(Benchmark):
33
 
34
  class ReluDtypeBenchmark(Benchmark):
35
  """Dtypes this build ships beyond upstream's fp32-only: fp16, bf16, int8.
36
- base = 1024^2, large = 4096^2, matching ReluBenchmark's shapes."""
 
 
 
 
 
 
 
 
 
37
 
38
  seed: int = 42
39
 
40
  def _float_setup(self, side, dtype):
41
  self.x = torch.randn(side, side, device=self.device, dtype=dtype)
42
  self.out = torch.empty_like(self.x)
 
43
 
44
  def setup_fp16_base(self):
45
  self._float_setup(1024, torch.float16)
@@ -58,6 +68,7 @@ class ReluDtypeBenchmark(Benchmark):
58
  -128, 128, (side, side), device=self.device, dtype=torch.int8
59
  )
60
  self.out = torch.empty_like(self.x)
 
61
 
62
  def setup_int8_base(self):
63
  self._int8_setup(1024)
@@ -66,37 +77,43 @@ class ReluDtypeBenchmark(Benchmark):
66
  self._int8_setup(4096)
67
 
68
  def benchmark_fp16_base(self):
69
- self.out = self.kernel.relu(self.x)
70
 
71
  def benchmark_fp16_large(self):
72
- self.out = self.kernel.relu(self.x)
73
 
74
  def benchmark_bf16_base(self):
75
- self.out = self.kernel.relu(self.x)
76
 
77
  def benchmark_bf16_large(self):
78
- self.out = self.kernel.relu(self.x)
79
 
80
  def benchmark_int8_base(self):
81
- self.out = self.kernel.relu(self.x)
82
 
83
  def benchmark_int8_large(self):
84
- self.out = self.kernel.relu(self.x)
85
 
86
  def verify_fp16_base(self) -> torch.Tensor:
87
- return F.relu(self.x)
 
88
 
89
  def verify_fp16_large(self) -> torch.Tensor:
90
- return F.relu(self.x)
 
91
 
92
  def verify_bf16_base(self) -> torch.Tensor:
93
- return F.relu(self.x)
 
94
 
95
  def verify_bf16_large(self) -> torch.Tensor:
96
- return F.relu(self.x)
 
97
 
98
  def verify_int8_base(self) -> torch.Tensor:
99
- return F.relu(self.x)
 
100
 
101
  def verify_int8_large(self) -> torch.Tensor:
102
- return F.relu(self.x)
 
 
33
 
34
  class ReluDtypeBenchmark(Benchmark):
35
  """Dtypes this build ships beyond upstream's fp32-only: fp16, bf16, int8.
36
+ base = 1024^2, large = 4096^2, matching ReluBenchmark's shapes.
37
+
38
+ Uses the kernel's out= form with a preallocated output: the allocating
39
+ form (`self.out = kernel.relu(x)`) keeps the previous output alive at
40
+ allocation time, so the caching allocator alternates two output blocks
41
+ and the effective footprint grows by a full buffer — at L2-boundary
42
+ working sets that alone pushes the timed loop off the L2 cliff while
43
+ the single-shot reference timing stays on it (torch.relu itself
44
+ measures ~0.33x vs its own single-shot under the same loop). The out=
45
+ form keeps one live output, matching the reference's footprint."""
46
 
47
  seed: int = 42
48
 
49
  def _float_setup(self, side, dtype):
50
  self.x = torch.randn(side, side, device=self.device, dtype=dtype)
51
  self.out = torch.empty_like(self.x)
52
+ self.ref_out = torch.empty_like(self.x)
53
 
54
  def setup_fp16_base(self):
55
  self._float_setup(1024, torch.float16)
 
68
  -128, 128, (side, side), device=self.device, dtype=torch.int8
69
  )
70
  self.out = torch.empty_like(self.x)
71
+ self.ref_out = torch.empty_like(self.x)
72
 
73
  def setup_int8_base(self):
74
  self._int8_setup(1024)
 
77
  self._int8_setup(4096)
78
 
79
  def benchmark_fp16_base(self):
80
+ self.kernel.relu(self.x, out=self.out)
81
 
82
  def benchmark_fp16_large(self):
83
+ self.kernel.relu(self.x, out=self.out)
84
 
85
  def benchmark_bf16_base(self):
86
+ self.kernel.relu(self.x, out=self.out)
87
 
88
  def benchmark_bf16_large(self):
89
+ self.kernel.relu(self.x, out=self.out)
90
 
91
  def benchmark_int8_base(self):
92
+ self.kernel.relu(self.x, out=self.out)
93
 
94
  def benchmark_int8_large(self):
95
+ self.kernel.relu(self.x, out=self.out)
96
 
97
  def verify_fp16_base(self) -> torch.Tensor:
98
+ torch.clamp(self.x, min=0, out=self.ref_out)
99
+ return self.ref_out
100
 
101
  def verify_fp16_large(self) -> torch.Tensor:
102
+ torch.clamp(self.x, min=0, out=self.ref_out)
103
+ return self.ref_out
104
 
105
  def verify_bf16_base(self) -> torch.Tensor:
106
+ torch.clamp(self.x, min=0, out=self.ref_out)
107
+ return self.ref_out
108
 
109
  def verify_bf16_large(self) -> torch.Tensor:
110
+ torch.clamp(self.x, min=0, out=self.ref_out)
111
+ return self.ref_out
112
 
113
  def verify_int8_base(self) -> torch.Tensor:
114
+ torch.clamp(self.x, min=0, out=self.ref_out)
115
+ return self.ref_out
116
 
117
  def verify_int8_large(self) -> torch.Tensor:
118
+ torch.clamp(self.x, min=0, out=self.ref_out)
119
+ return self.ref_out
build.toml CHANGED
@@ -1,7 +1,7 @@
1
  [general]
2
  name = "relu"
3
  license = "Apache-2.0"
4
- version = 2
5
  backends = [
6
  "cpu",
7
  "cuda",
 
1
  [general]
2
  name = "relu"
3
  license = "Apache-2.0"
4
+ version = 3
5
  backends = [
6
  "cpu",
7
  "cuda",
build/torch212-cxx11-cu126-x86_64-linux/_ops.py CHANGED
@@ -1,9 +1,9 @@
1
  import torch
2
- from . import _relu_cuda_f03b26d
3
- ops = torch.ops._relu_cuda_f03b26d
4
 
5
  def add_op_namespace_prefix(op_name: str):
6
  """
7
  Prefix op by namespace.
8
  """
9
- return f"_relu_cuda_f03b26d::{op_name}"
 
1
  import torch
2
+ from . import _relu_cuda_3c4de86
3
+ ops = torch.ops._relu_cuda_3c4de86
4
 
5
  def add_op_namespace_prefix(op_name: str):
6
  """
7
  Prefix op by namespace.
8
  """
9
+ return f"_relu_cuda_3c4de86::{op_name}"
build/torch212-cxx11-cu126-x86_64-linux/{_relu_cuda_f03b26d.abi3.so → _relu_cuda_3c4de86.abi3.so} RENAMED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:b63a2a582b659c860f7fa7fd98e5a958c605c2e20447719f41a40782c0ad9a9c
3
- size 375528
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9e051f8ac8a1378546c97f96e66429a16f476d1bdfeedb4d9a563a06528ce2d2
3
+ size 457720
build/torch212-cxx11-cu126-x86_64-linux/metadata.json CHANGED
@@ -1,7 +1,7 @@
1
  {
2
  "name": "relu",
3
- "id": "_relu_cuda_f03b26d",
4
- "version": 2,
5
  "license": "Apache-2.0",
6
  "python-depends": [],
7
  "backend": {
@@ -21,8 +21,8 @@
21
  "algorithm": "sha256",
22
  "files": {
23
  "__init__.py": "lzFVikVVZVGdTURXq9skL4x50/8QcWTs99kuLNnXtG8=",
24
- "_ops.py": "Ak/47NpRYMI1BdAvxs4t0BOxt8lTylWNqE7GkOusUIE=",
25
- "_relu_cuda_f03b26d.abi3.so": "tjoqWCtlnIYPf6f9mOWpWMYFwuIER3GfQaQHgsCtmpw=",
26
  "relu/__init__.py": "DFYPlrhXwYjEqCl/8n0SmWGZV8NFml5DPhMjKfv98GY="
27
  }
28
  }
 
1
  {
2
  "name": "relu",
3
+ "id": "_relu_cuda_3c4de86",
4
+ "version": 3,
5
  "license": "Apache-2.0",
6
  "python-depends": [],
7
  "backend": {
 
21
  "algorithm": "sha256",
22
  "files": {
23
  "__init__.py": "lzFVikVVZVGdTURXq9skL4x50/8QcWTs99kuLNnXtG8=",
24
+ "_ops.py": "0wm/ZKMPsoPIX+o9zUpD81Z04dcjtHEbfq+UDPEdACs=",
25
+ "_relu_cuda_3c4de86.abi3.so": "ngUfisihN4VGyX+W5mQpoW9HbRvf7ttNmlY6BlKM4tI=",
26
  "relu/__init__.py": "DFYPlrhXwYjEqCl/8n0SmWGZV8NFml5DPhMjKfv98GY="
27
  }
28
  }
relu_cuda/relu.cu CHANGED
@@ -12,9 +12,10 @@
12
  // ReLU is memory-bound, so the design is what the config sweeps proved optimal:
13
  // * 128-bit (int4) vectorized loads/stores -> 16/sizeof(T) elements per thread.
14
  // * Streaming cache hints (__ldcs / __stcs) for write-once/read-once data.
15
- // * Arch-adaptive launch (see launch_relu): blk128/512 + ld+st.cs everywhere,
16
- // except Hopper+ below a 16 MB working set, where stream-store-only
17
- // unroll-2 blk256 measured faster in-harness.
 
18
  // Scalar tail for numel not divisible by the vec width.
19
  // Supported dtypes: float32, float16, bfloat16, int8. Element throughput scales
20
  // ~1/sizeof: fp16/bf16 ~2x fp32, int8 ~4x fp32 (all at the same ~945 GB/s wall).
@@ -95,22 +96,35 @@ static void launch_relu(torch::Tensor &out, torch::Tensor const &input,
95
 
96
  // Base config (swept on RTX 4090, bench/sweep.cu): 128-thread blocks when
97
  // L2-resident, 512 when DRAM-bound (32 MB = measured crossover), streaming
98
- // loads and stores throughout.
99
  int threads = (working_set < (int64_t{32} << 20)) ? 128 : 512;
100
  int unroll = 1;
101
  bool ldcs = true;
102
 
103
- // Hopper+ small-working-set override. Of the standalone H100 sweep
104
- // (bench/sweep_h100.cu), only this part survived in-harness verification
105
- // (nsys interleaved, robust_bench): stream-store-only unroll-2 block-256 is
106
- // +4.6% at 8.4 MB, but -27% at 33.6 MB under realistic L2 contention (the
107
- // standalone sweep's +16-20% there did not reproduce, and neither did its
108
- // DRAM-side blk1024 gain: 0 to -0.8% in-harness). 16 MB is a conservative
109
- // cut between the verified 8.4 MB win and the verified 33.6 MB loss.
110
  // Cached device props from torch -- no per-call driver query.
111
- if (at::cuda::getCurrentDeviceProperties()->major >= 9 &&
112
- working_set < (int64_t{16} << 20)) {
113
- threads = 256; unroll = 2; ldcs = false;
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
114
  }
115
 
116
  if (nvec > 0) {
@@ -121,6 +135,8 @@ static void launch_relu(torch::Tensor &out, torch::Tensor const &input,
121
  auto const *i = reinterpret_cast<int4 const *>(ip);
122
  if (unroll == 2 && !ldcs)
123
  relu_vec<T, 2, false, true><<<blocks, threads, 0, stream>>>(o, i, nvec);
 
 
124
  else
125
  relu_vec<T, 1, true, true><<<blocks, threads, 0, stream>>>(o, i, nvec);
126
  }
 
12
  // ReLU is memory-bound, so the design is what the config sweeps proved optimal:
13
  // * 128-bit (int4) vectorized loads/stores -> 16/sizeof(T) elements per thread.
14
  // * Streaming cache hints (__ldcs / __stcs) for write-once/read-once data.
15
+ // * Arch-adaptive launch (see launch_relu): blk128/512 + ld+st.cs, except
16
+ // (a) pre-Hopper working sets in (32 MiB, L2], which use default loads +
17
+ // streaming stores (L2_BAND_CLIFF.md), and (b) Hopper+ below 16 MB, which
18
+ // uses stream-store-only unroll-2 blk256.
19
  // Scalar tail for numel not divisible by the vec width.
20
  // Supported dtypes: float32, float16, bfloat16, int8. Element throughput scales
21
  // ~1/sizeof: fp16/bf16 ~2x fp32, int8 ~4x fp32 (all at the same ~945 GB/s wall).
 
96
 
97
  // Base config (swept on RTX 4090, bench/sweep.cu): 128-thread blocks when
98
  // L2-resident, 512 when DRAM-bound (32 MB = measured crossover), streaming
99
+ // loads and stores.
100
  int threads = (working_set < (int64_t{32} << 20)) ? 128 : 512;
101
  int unroll = 1;
102
  bool ldcs = true;
103
 
 
 
 
 
 
 
 
104
  // Cached device props from torch -- no per-call driver query.
105
+ const cudaDeviceProp *prop = at::cuda::getCurrentDeviceProperties();
106
+ if (prop->major >= 9) {
107
+ // Hopper+ small-working-set override. Of the standalone H100 sweep
108
+ // (bench/sweep_h100.cu), only this part survived in-harness verification:
109
+ // stream-store-only unroll-2 block-256, +4.6-9.3% below 16 MB. Hopper's
110
+ // evict-first retention is strong enough that the pre-Hopper L2-band
111
+ // cliff below does NOT occur there (measured: 1.43-1.55x over torch at
112
+ // 32-50 MiB working sets with the streaming config).
113
+ if (working_set < (int64_t{16} << 20)) {
114
+ threads = 256; unroll = 2; ldcs = false;
115
+ }
116
+ } else if (working_set >= (int64_t{32} << 20) &&
117
+ working_set <= static_cast<int64_t>(prop->l2CacheSize)) {
118
+ // Pre-Hopper L2-band fix (independently reported, reproduced, and
119
+ // root-caused 2026-07-31; see L2_BAND_CLIFF.md):
120
+ // with evict-first loads, working sets in (32 MiB, L2] run 2-3x SLOWER
121
+ // than torch.relu -- __ldcs discards exactly the input-reuse residency
122
+ // torch exploits (measured 0.34-0.65x vs torch across the band on the
123
+ // 4090's 72 MiB L2, for every dtype at its own byte sizes). Default
124
+ // loads restore parity-to-better; stores stay streaming (output is
125
+ // write-once -- evicting it early keeps more L2 for the input). Above
126
+ // L2, streaming loads win again (1.13-1.30x vs torch) and stay.
127
+ ldcs = false;
128
  }
129
 
130
  if (nvec > 0) {
 
135
  auto const *i = reinterpret_cast<int4 const *>(ip);
136
  if (unroll == 2 && !ldcs)
137
  relu_vec<T, 2, false, true><<<blocks, threads, 0, stream>>>(o, i, nvec);
138
+ else if (!ldcs)
139
+ relu_vec<T, 1, false, true><<<blocks, threads, 0, stream>>>(o, i, nvec);
140
  else
141
  relu_vec<T, 1, true, true><<<blocks, threads, 0, stream>>>(o, i, nvec);
142
  }