liangsu9988 commited on
Commit
99e17c0
·
verified ·
1 Parent(s): 991010b

Promote latest kernel artifacts to main

Browse files
README.md CHANGED
@@ -1,9 +1,135 @@
1
- # flashrt/fp8-gemm
2
 
3
- This repository is a compatibility mirror for older `kernels` clients
4
- that resolve repositories through the default Hugging Face model repo API.
5
 
6
- Canonical Kernel Hub repo: https://huggingface.co/kernels/flashrt/fp8-gemm
 
 
 
7
 
8
- Do not edit this mirror by hand. It is generated from the Kernel Hub
9
- `vN` branches and contains the same `build/**` artifacts.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # fp8-gemm
2
 
3
+ FlashRT native CUDA FP8 GEMV/GEMM kernels for low-latency transformer and
4
+ diffuser linear layers on NVIDIA Ada SM89 and Blackwell SM110/SM120 GPUs.
5
 
6
+ This package exposes the hand-tuned FP8 E4M3 decode and small-M kernels as
7
+ Tensor APIs for Hugging Face Kernel Hub. It is intended for model runtimes that
8
+ already hold activations and weights in FP8 and want a low-overhead BF16 output
9
+ linear path.
10
 
11
+ ## Available Functions
12
+
13
+ - `fp8_linear_bf16(input, weight, alpha=1.0, out=None, variant=0)`
14
+ - `fp8_linear_residual_bf16(input, weight, residual, alpha=1.0, variant=0)`
15
+ - `fp8_linear_bias_bf16(input, weight, bias, alpha=1.0, out=None)`
16
+ - `fp8_linear_bias_residual_bf16(input, weight, bias, residual, alpha=1.0)`
17
+ - `fp8_linear_bias_gelu_bf16(input, weight, bias, alpha=1.0, out=None)`
18
+ - `fp8_blockwise_linear_bf16(input, weight, input_scale, weight_scale, out=None)`
19
+ - `fp8_blockwise_swiglu_quantize_fp8(input, gate_up_weight, input_scale, gate_up_weight_scale, output=None, output_scale=None)`
20
+ - `select_fp8_linear_tile(m, n, k, variant=0)`
21
+
22
+ Tensor contract:
23
+
24
+ - `input`: `torch.float8_e4m3fn`, shape `(M, K)`, contiguous CUDA tensor.
25
+ - `weight`: `torch.float8_e4m3fn`, shape `(N, K)`, contiguous CUDA tensor.
26
+ - `out`: `torch.bfloat16`, shape `(M, N)`.
27
+ - `residual`: `torch.bfloat16`, shape `(1, N)` or `(N,)`, only supported for
28
+ the `M=1` decode GEMV path.
29
+ - `K % 16 == 0`; SM120 additionally requires `K % 32 == 0`.
30
+ - On SM120, `M == 1` uses dedicated GEMV and `2 <= M <= 64` uses small-M
31
+ GEMM tiles.
32
+ - On SM110 (Jetson AGX Thor), the per-tensor API uses the production FlashRT
33
+ CUTLASS Sq/T1/Wide family and supports the validated model-shape matrix from
34
+ decode through large vision/backbone rows. The large-M production band is
35
+ validated from `M=65` through `M=1024`, including PI0.5 prefill QKV, O,
36
+ gate/up, and down projections at `M=712..970`. `N` and `K` must be divisible
37
+ by 16.
38
+ - The three BF16 bias APIs are SM110-only. They accept BF16 `(N,)` bias and
39
+ preserve the same row-major FP8 `(M,K)` input and `(N,K)` weight contract.
40
+ The residual API updates a BF16 `(M,N)` tensor in place. The GELU API uses
41
+ the tanh approximation.
42
+ - SM110 `variant=0` is the production auto dispatcher. Diagnostic variants are
43
+ `1=Sq`, `2=T1`, and `3=Wide`; they are correctness-tested but should not be
44
+ pinned by model integrations without a shape-specific benchmark.
45
+ - The per-tensor kernels use Blackwell FP8 MMA instructions and are not valid
46
+ for SM89. SM89 support is provided by the blockwise API below.
47
+ - `alpha` is a host float. For per-tensor FP8 quantization, pass
48
+ `float(input_scale * weight_scale)` from your static calibration metadata.
49
+
50
+ The blockwise API uses a separate contract:
51
+
52
+ - `input`: FP8 E4M3 `(M, K)`.
53
+ - `weight`: FP8 E4M3 `(N, K)`.
54
+ - `input_scale`: FP32 `(M, K / 128)`.
55
+ - `weight_scale`: FP32 `(N / 128, K / 128)`.
56
+ - `N` and `K` must be divisible by 128; `M` is unrestricted.
57
+ - Output is BF16 `(M, N)`.
58
+ - On SM89, the blockwise API dispatches to the production FlashRT native
59
+ `mma.sync.aligned.m16n8k32` GEMM/GEMV implementation.
60
+ - On SM120, it dispatches to the production FlashRT CUTLASS block-scaled
61
+ implementation.
62
+ - SM110 is intentionally not claimed by the blockwise API; use the per-tensor
63
+ static-scale path there. Other architectures are rejected explicitly.
64
+
65
+ The fused SM89 producer accepts FP8 `(M,K)` input, FP8 `(2*N,K)` gate/up
66
+ weight, block-128 FP32 scales, and returns FP8 `(M,N)` plus FP32 `(M,N/128)`
67
+ output scales. Its public range is `1 <= M <= 256` with `N` and `K` divisible
68
+ by 128. It is rejected explicitly on non-SM89 GPUs.
69
+
70
+ ## Minimal Usage
71
+
72
+ ```python
73
+ from kernels import get_kernel
74
+ import torch
75
+
76
+ ops = get_kernel("flashrt/fp8-gemm", version=1, trust_remote_code=True)
77
+
78
+ x = torch.randn((16, 4096), device="cuda", dtype=torch.bfloat16).to(torch.float8_e4m3fn)
79
+ w = torch.randn((8192, 4096), device="cuda", dtype=torch.bfloat16).to(torch.float8_e4m3fn)
80
+
81
+ y = ops.fp8_linear_bf16(x, w, alpha=1.0)
82
+ ```
83
+
84
+ SM110 bias epilogues:
85
+
86
+ ```python
87
+ bias = torch.randn((8192,), device="cuda", dtype=torch.bfloat16)
88
+ residual = torch.randn((16, 8192), device="cuda", dtype=torch.bfloat16)
89
+
90
+ y = ops.fp8_linear_bias_bf16(x, w, bias, alpha=1.0)
91
+ ops.fp8_linear_bias_residual_bf16(x, w, bias, residual, alpha=1.0)
92
+ y_gelu = ops.fp8_linear_bias_gelu_bf16(x, w, bias, alpha=1.0)
93
+ ```
94
+
95
+ Warm each distinct SM110 bias shape once before CUDA Graph capture. The
96
+ cuBLASLt fallback lazily creates and caches its descriptor, algorithm, and
97
+ workspace on the first call; replay itself performs no allocation.
98
+
99
+ Decode residual path:
100
+
101
+ ```python
102
+ x = torch.randn((1, 4096), device="cuda", dtype=torch.bfloat16).to(torch.float8_e4m3fn)
103
+ w = torch.randn((4096, 4096), device="cuda", dtype=torch.bfloat16).to(torch.float8_e4m3fn)
104
+ residual = torch.zeros((1, 4096), device="cuda", dtype=torch.bfloat16)
105
+
106
+ ops.fp8_linear_residual_bf16(x, w, residual, alpha=1.0)
107
+ ```
108
+
109
+ Block-128 scaling:
110
+
111
+ ```python
112
+ m, k, n = 51, 1536, 1536
113
+ x = torch.randn((m, k), device="cuda").to(torch.float8_e4m3fn)
114
+ w = torch.randn((n, k), device="cuda").to(torch.float8_e4m3fn)
115
+ x_scale = torch.ones((m, k // 128), device="cuda", dtype=torch.float32)
116
+ w_scale = torch.ones((n // 128, k // 128), device="cuda", dtype=torch.float32)
117
+
118
+ y = ops.fp8_blockwise_linear_bf16(x, w, x_scale, w_scale)
119
+ ```
120
+
121
+ ## Validation
122
+
123
+ ```bash
124
+ python fp8-gemm/tests/test_fp8_gemm.py --backend source --mode full
125
+ python fp8-gemm/benchmarks/benchmark.py --backend source --mode headline
126
+ python fp8-gemm/benchmarks/benchmark.py --backend source --mode pi05-prefill
127
+ python fp8-gemm/benchmarks/benchmark_bias.py --backend source
128
+ ```
129
+
130
+ The SM110 full sweep covers PI0.5, GROOT N1.6/N1.7, Cosmos Edge, and LingBot
131
+ VLA projection families, plus decode, generic small-M, the `M=65` large-M
132
+ boundary, and the three SigLIP bias epilogues. Public
133
+ benchmark tables are only updated after source correctness, installed artifact
134
+ correctness, shape/tile sweeps, `torch.compile(fullgraph=True)`, CUDA Graph
135
+ replay, and parity against the original FlashRT native pointer entry pass.
build/torch211-cxx11-cu128-x86_64-linux/__init__.py CHANGED
@@ -37,6 +37,34 @@ def _fp8_linear_residual_bf16_fake(
37
  return None
38
 
39
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
40
  @torch.library.register_fake(add_op_namespace_prefix("fp8_blockwise_linear_bf16"))
41
  def _fp8_blockwise_linear_bf16_fake(
42
  input: torch.Tensor,
@@ -95,9 +123,35 @@ def select_fp8_linear_tile(m: int, n: int, k: int, variant: int = 0) -> str:
95
  variant = int(variant)
96
  if m <= 0 or n <= 0 or k <= 0:
97
  raise RuntimeError("m, n, and k must be positive")
98
- if k % 32 != 0:
99
- raise RuntimeError("k must be divisible by 32")
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
100
  if m == 1:
 
 
101
  if variant == 4:
102
  return "gemv_fp8_m1_w4"
103
  if variant == 8:
@@ -113,6 +167,8 @@ def select_fp8_linear_tile(m: int, n: int, k: int, variant: int = 0) -> str:
113
  return "gemv_fp8_m1_w16"
114
  if variant != 0:
115
  raise RuntimeError("small-M dispatcher currently supports variant=0 only")
 
 
116
  if m <= 16:
117
  if k % 256 == 0:
118
  return "ld_fp8_gemm_16x128x256_w4" if n % 128 == 0 else "ld_fp8_gemm_16x64x256_w4"
@@ -151,7 +207,9 @@ def fp8_linear_bf16(
151
 
152
  ``input`` and ``weight`` must be FP8 E4M3 CUDA tensors with shapes
153
  ``(M, K)`` and ``(N, K)``. ``alpha`` is a host float, normally the product
154
- of static per-tensor input and weight scales.
 
 
155
  """
156
 
157
  if out is None:
@@ -177,6 +235,56 @@ def fp8_linear_residual_bf16(
177
  return residual
178
 
179
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
180
  def fp8_blockwise_linear_bf16(
181
  input: torch.Tensor,
182
  weight: torch.Tensor,
@@ -184,7 +292,7 @@ def fp8_blockwise_linear_bf16(
184
  weight_scale: torch.Tensor,
185
  out: torch.Tensor | None = None,
186
  ) -> torch.Tensor:
187
- """Block-128 scaled FP8 linear with BF16 output on SM120."""
188
 
189
  if out is None:
190
  out = torch.empty(
@@ -228,6 +336,9 @@ def fp8_blockwise_swiglu_quantize_fp8(
228
  __all__ = [
229
  "fp8_linear_bf16",
230
  "fp8_linear_residual_bf16",
 
 
 
231
  "fp8_blockwise_linear_bf16",
232
  "fp8_blockwise_swiglu_quantize_fp8",
233
  "select_fp8_linear_tile",
 
37
  return None
38
 
39
 
40
+ def _check_bias_linear_shapes(input, weight, bias, out) -> None:
41
+ if input.dim() != 2 or weight.dim() != 2:
42
+ raise RuntimeError("input and weight must be rank-2 tensors")
43
+ if input.shape[1] != weight.shape[1]:
44
+ raise RuntimeError("input and weight K dimensions must match")
45
+ if bias.shape != (weight.shape[0],):
46
+ raise RuntimeError("bias must have shape (weight.shape[0],)")
47
+ if out.shape != (input.shape[0], weight.shape[0]):
48
+ raise RuntimeError("out must have shape (input.shape[0], weight.shape[0])")
49
+
50
+
51
+ @torch.library.register_fake(add_op_namespace_prefix("fp8_linear_bias_bf16"))
52
+ def _fp8_linear_bias_bf16_fake(input, weight, bias, alpha: float, out) -> None:
53
+ _check_bias_linear_shapes(input, weight, bias, out)
54
+
55
+
56
+ @torch.library.register_fake(add_op_namespace_prefix("fp8_linear_bias_residual_bf16"))
57
+ def _fp8_linear_bias_residual_bf16_fake(
58
+ input, weight, bias, alpha: float, residual
59
+ ) -> None:
60
+ _check_bias_linear_shapes(input, weight, bias, residual)
61
+
62
+
63
+ @torch.library.register_fake(add_op_namespace_prefix("fp8_linear_bias_gelu_bf16"))
64
+ def _fp8_linear_bias_gelu_bf16_fake(input, weight, bias, alpha: float, out) -> None:
65
+ _check_bias_linear_shapes(input, weight, bias, out)
66
+
67
+
68
  @torch.library.register_fake(add_op_namespace_prefix("fp8_blockwise_linear_bf16"))
69
  def _fp8_blockwise_linear_bf16_fake(
70
  input: torch.Tensor,
 
123
  variant = int(variant)
124
  if m <= 0 or n <= 0 or k <= 0:
125
  raise RuntimeError("m, n, and k must be positive")
126
+ if k % 16 != 0:
127
+ raise RuntimeError("k must be divisible by 16")
128
+ capability = torch.cuda.get_device_capability() if torch.cuda.is_available() else None
129
+ if capability == (11, 0):
130
+ forced = {1: "sm110_sq_bf16", 2: "sm110_t1_bf16", 3: "sm110_wide_bf16"}
131
+ if variant not in {0, *forced}:
132
+ raise RuntimeError("SM110 variant must be 0 (auto), 1 (Sq), 2 (T1), or 3 (Wide)")
133
+ if n % 16 or k % 16:
134
+ raise RuntimeError("SM110 requires n and k divisible by 16")
135
+ if variant:
136
+ return forced[variant]
137
+ if m >= 512 and k == 2048 and 2048 <= n <= 2560:
138
+ return "sm110_sq_bf16"
139
+ if m >= 512 and n >= 16 * k:
140
+ return "sm110_t1_bf16"
141
+ if m >= 512 and k >= 4 * n:
142
+ return "sm110_wide_bf16"
143
+ if n >= 8 * k:
144
+ return "sm110_wide_bf16"
145
+ if m >= 128 and k >= 4 * n:
146
+ return "sm110_sq_bf16"
147
+ if n == k and m >= 512:
148
+ return "sm110_sq_bf16" if k <= 1024 else "sm110_wide_bf16"
149
+ if n == k and m >= 128:
150
+ return "sm110_wide_bf16"
151
+ return "sm110_t1_bf16"
152
  if m == 1:
153
+ if k % 32:
154
+ raise RuntimeError("SM120 requires k divisible by 32")
155
  if variant == 4:
156
  return "gemv_fp8_m1_w4"
157
  if variant == 8:
 
167
  return "gemv_fp8_m1_w16"
168
  if variant != 0:
169
  raise RuntimeError("small-M dispatcher currently supports variant=0 only")
170
+ if k % 32:
171
+ raise RuntimeError("SM120 requires k divisible by 32")
172
  if m <= 16:
173
  if k % 256 == 0:
174
  return "ld_fp8_gemm_16x128x256_w4" if n % 128 == 0 else "ld_fp8_gemm_16x64x256_w4"
 
207
 
208
  ``input`` and ``weight`` must be FP8 E4M3 CUDA tensors with shapes
209
  ``(M, K)`` and ``(N, K)``. ``alpha`` is a host float, normally the product
210
+ of static per-tensor input and weight scales. SM110 uses the production
211
+ CUTLASS Sq/T1/Wide dispatcher over full model row counts; SM120 uses the
212
+ hand-tuned M<=64 path.
213
  """
214
 
215
  if out is None:
 
235
  return residual
236
 
237
 
238
+ def fp8_linear_bias_bf16(
239
+ input: torch.Tensor,
240
+ weight: torch.Tensor,
241
+ bias: torch.Tensor,
242
+ alpha: float = 1.0,
243
+ out: torch.Tensor | None = None,
244
+ ) -> torch.Tensor:
245
+ """SM110 FP8 linear with fused BF16 bias and BF16 output."""
246
+ if out is None:
247
+ out = torch.empty(
248
+ (input.shape[0], weight.shape[0]),
249
+ device=input.device,
250
+ dtype=torch.bfloat16,
251
+ )
252
+ ops.fp8_linear_bias_bf16(input, weight, bias, float(alpha), out)
253
+ return out
254
+
255
+
256
+ def fp8_linear_bias_residual_bf16(
257
+ input: torch.Tensor,
258
+ weight: torch.Tensor,
259
+ bias: torch.Tensor,
260
+ residual: torch.Tensor,
261
+ alpha: float = 1.0,
262
+ ) -> torch.Tensor:
263
+ """SM110 fused ``residual += alpha * input @ weight.T + bias``."""
264
+ ops.fp8_linear_bias_residual_bf16(
265
+ input, weight, bias, float(alpha), residual
266
+ )
267
+ return residual
268
+
269
+
270
+ def fp8_linear_bias_gelu_bf16(
271
+ input: torch.Tensor,
272
+ weight: torch.Tensor,
273
+ bias: torch.Tensor,
274
+ alpha: float = 1.0,
275
+ out: torch.Tensor | None = None,
276
+ ) -> torch.Tensor:
277
+ """SM110 FP8 linear with fused BF16 bias and GELU epilogue."""
278
+ if out is None:
279
+ out = torch.empty(
280
+ (input.shape[0], weight.shape[0]),
281
+ device=input.device,
282
+ dtype=torch.bfloat16,
283
+ )
284
+ ops.fp8_linear_bias_gelu_bf16(input, weight, bias, float(alpha), out)
285
+ return out
286
+
287
+
288
  def fp8_blockwise_linear_bf16(
289
  input: torch.Tensor,
290
  weight: torch.Tensor,
 
292
  weight_scale: torch.Tensor,
293
  out: torch.Tensor | None = None,
294
  ) -> torch.Tensor:
295
+ """Block-128 scaled FP8 linear with BF16 output on SM89/SM120."""
296
 
297
  if out is None:
298
  out = torch.empty(
 
336
  __all__ = [
337
  "fp8_linear_bf16",
338
  "fp8_linear_residual_bf16",
339
+ "fp8_linear_bias_bf16",
340
+ "fp8_linear_bias_residual_bf16",
341
+ "fp8_linear_bias_gelu_bf16",
342
  "fp8_blockwise_linear_bf16",
343
  "fp8_blockwise_swiglu_quantize_fp8",
344
  "select_fp8_linear_tile",
build/torch211-cxx11-cu128-x86_64-linux/{_fp8_gemm_cuda_bcb0782.abi3.so → _fp8_gemm_cuda_a6f59d3.abi3.so} RENAMED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:182510f121f814d7c367c58c641dc57f8992cc1b62fa25409ed9128e292fcec8
3
- size 3942856
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:050b9317e0055fff914a3cf9ff3177bc890b3f29f549780f5feb96e406b58093
3
+ size 3968152
build/torch211-cxx11-cu128-x86_64-linux/_ops.py CHANGED
@@ -1,9 +1,9 @@
1
  import torch
2
- from . import _fp8_gemm_cuda_bcb0782
3
- ops = torch.ops._fp8_gemm_cuda_bcb0782
4
 
5
  def add_op_namespace_prefix(op_name: str):
6
  """
7
  Prefix op by namespace.
8
  """
9
- return f"_fp8_gemm_cuda_bcb0782::{op_name}"
 
1
  import torch
2
+ from . import _fp8_gemm_cuda_a6f59d3
3
+ ops = torch.ops._fp8_gemm_cuda_a6f59d3
4
 
5
  def add_op_namespace_prefix(op_name: str):
6
  """
7
  Prefix op by namespace.
8
  """
9
+ return f"_fp8_gemm_cuda_a6f59d3::{op_name}"
build/torch211-cxx11-cu128-x86_64-linux/metadata.json CHANGED
@@ -1,6 +1,6 @@
1
  {
2
  "name": "fp8-gemm",
3
- "id": "_fp8_gemm_cuda_bcb0782",
4
  "version": 1,
5
  "license": "Apache-2.0",
6
  "python-depends": [],
@@ -14,9 +14,9 @@
14
  "digest": {
15
  "algorithm": "sha256",
16
  "files": {
17
- "__init__.py": "AhlkcnyTuk3G4aYsvMqu3QdiEZSTA6Y00poqqiX+Mps=",
18
- "_fp8_gemm_cuda_bcb0782.abi3.so": "GCUQ8SH4FNfDZ8WMZB3Ff4mSzBti+iVAntkSjikvzsg=",
19
- "_ops.py": "qpBYMOKsfVJtHhKzoX/hxuQPF3MGISv+MQwpPmbzUcw=",
20
  "fp8_gemm/__init__.py": "DFYPlrhXwYjEqCl/8n0SmWGZV8NFml5DPhMjKfv98GY="
21
  }
22
  }
 
1
  {
2
  "name": "fp8-gemm",
3
+ "id": "_fp8_gemm_cuda_a6f59d3",
4
  "version": 1,
5
  "license": "Apache-2.0",
6
  "python-depends": [],
 
14
  "digest": {
15
  "algorithm": "sha256",
16
  "files": {
17
+ "__init__.py": "HsFfsGY7AK/6Fziv4r2VQ+R2kDjS6hxKKgWqlA+zrKY=",
18
+ "_fp8_gemm_cuda_a6f59d3.abi3.so": "BQuTF+AFX/+RSjz5/zF3vIkLPyn1SXgPX+uW5Aa1gJM=",
19
+ "_ops.py": "kgQ6LWi/1cNaF3rEiBHA8t3jyz0toqRVj94wVDPiqVU=",
20
  "fp8_gemm/__init__.py": "DFYPlrhXwYjEqCl/8n0SmWGZV8NFml5DPhMjKfv98GY="
21
  }
22
  }
build/torch211-cxx11-cu130-x86_64-linux/__init__.py CHANGED
@@ -37,6 +37,34 @@ def _fp8_linear_residual_bf16_fake(
37
  return None
38
 
39
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
40
  @torch.library.register_fake(add_op_namespace_prefix("fp8_blockwise_linear_bf16"))
41
  def _fp8_blockwise_linear_bf16_fake(
42
  input: torch.Tensor,
@@ -95,9 +123,35 @@ def select_fp8_linear_tile(m: int, n: int, k: int, variant: int = 0) -> str:
95
  variant = int(variant)
96
  if m <= 0 or n <= 0 or k <= 0:
97
  raise RuntimeError("m, n, and k must be positive")
98
- if k % 32 != 0:
99
- raise RuntimeError("k must be divisible by 32")
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
100
  if m == 1:
 
 
101
  if variant == 4:
102
  return "gemv_fp8_m1_w4"
103
  if variant == 8:
@@ -113,6 +167,8 @@ def select_fp8_linear_tile(m: int, n: int, k: int, variant: int = 0) -> str:
113
  return "gemv_fp8_m1_w16"
114
  if variant != 0:
115
  raise RuntimeError("small-M dispatcher currently supports variant=0 only")
 
 
116
  if m <= 16:
117
  if k % 256 == 0:
118
  return "ld_fp8_gemm_16x128x256_w4" if n % 128 == 0 else "ld_fp8_gemm_16x64x256_w4"
@@ -151,7 +207,9 @@ def fp8_linear_bf16(
151
 
152
  ``input`` and ``weight`` must be FP8 E4M3 CUDA tensors with shapes
153
  ``(M, K)`` and ``(N, K)``. ``alpha`` is a host float, normally the product
154
- of static per-tensor input and weight scales.
 
 
155
  """
156
 
157
  if out is None:
@@ -177,6 +235,56 @@ def fp8_linear_residual_bf16(
177
  return residual
178
 
179
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
180
  def fp8_blockwise_linear_bf16(
181
  input: torch.Tensor,
182
  weight: torch.Tensor,
@@ -184,7 +292,7 @@ def fp8_blockwise_linear_bf16(
184
  weight_scale: torch.Tensor,
185
  out: torch.Tensor | None = None,
186
  ) -> torch.Tensor:
187
- """Block-128 scaled FP8 linear with BF16 output on SM120."""
188
 
189
  if out is None:
190
  out = torch.empty(
@@ -228,6 +336,9 @@ def fp8_blockwise_swiglu_quantize_fp8(
228
  __all__ = [
229
  "fp8_linear_bf16",
230
  "fp8_linear_residual_bf16",
 
 
 
231
  "fp8_blockwise_linear_bf16",
232
  "fp8_blockwise_swiglu_quantize_fp8",
233
  "select_fp8_linear_tile",
 
37
  return None
38
 
39
 
40
+ def _check_bias_linear_shapes(input, weight, bias, out) -> None:
41
+ if input.dim() != 2 or weight.dim() != 2:
42
+ raise RuntimeError("input and weight must be rank-2 tensors")
43
+ if input.shape[1] != weight.shape[1]:
44
+ raise RuntimeError("input and weight K dimensions must match")
45
+ if bias.shape != (weight.shape[0],):
46
+ raise RuntimeError("bias must have shape (weight.shape[0],)")
47
+ if out.shape != (input.shape[0], weight.shape[0]):
48
+ raise RuntimeError("out must have shape (input.shape[0], weight.shape[0])")
49
+
50
+
51
+ @torch.library.register_fake(add_op_namespace_prefix("fp8_linear_bias_bf16"))
52
+ def _fp8_linear_bias_bf16_fake(input, weight, bias, alpha: float, out) -> None:
53
+ _check_bias_linear_shapes(input, weight, bias, out)
54
+
55
+
56
+ @torch.library.register_fake(add_op_namespace_prefix("fp8_linear_bias_residual_bf16"))
57
+ def _fp8_linear_bias_residual_bf16_fake(
58
+ input, weight, bias, alpha: float, residual
59
+ ) -> None:
60
+ _check_bias_linear_shapes(input, weight, bias, residual)
61
+
62
+
63
+ @torch.library.register_fake(add_op_namespace_prefix("fp8_linear_bias_gelu_bf16"))
64
+ def _fp8_linear_bias_gelu_bf16_fake(input, weight, bias, alpha: float, out) -> None:
65
+ _check_bias_linear_shapes(input, weight, bias, out)
66
+
67
+
68
  @torch.library.register_fake(add_op_namespace_prefix("fp8_blockwise_linear_bf16"))
69
  def _fp8_blockwise_linear_bf16_fake(
70
  input: torch.Tensor,
 
123
  variant = int(variant)
124
  if m <= 0 or n <= 0 or k <= 0:
125
  raise RuntimeError("m, n, and k must be positive")
126
+ if k % 16 != 0:
127
+ raise RuntimeError("k must be divisible by 16")
128
+ capability = torch.cuda.get_device_capability() if torch.cuda.is_available() else None
129
+ if capability == (11, 0):
130
+ forced = {1: "sm110_sq_bf16", 2: "sm110_t1_bf16", 3: "sm110_wide_bf16"}
131
+ if variant not in {0, *forced}:
132
+ raise RuntimeError("SM110 variant must be 0 (auto), 1 (Sq), 2 (T1), or 3 (Wide)")
133
+ if n % 16 or k % 16:
134
+ raise RuntimeError("SM110 requires n and k divisible by 16")
135
+ if variant:
136
+ return forced[variant]
137
+ if m >= 512 and k == 2048 and 2048 <= n <= 2560:
138
+ return "sm110_sq_bf16"
139
+ if m >= 512 and n >= 16 * k:
140
+ return "sm110_t1_bf16"
141
+ if m >= 512 and k >= 4 * n:
142
+ return "sm110_wide_bf16"
143
+ if n >= 8 * k:
144
+ return "sm110_wide_bf16"
145
+ if m >= 128 and k >= 4 * n:
146
+ return "sm110_sq_bf16"
147
+ if n == k and m >= 512:
148
+ return "sm110_sq_bf16" if k <= 1024 else "sm110_wide_bf16"
149
+ if n == k and m >= 128:
150
+ return "sm110_wide_bf16"
151
+ return "sm110_t1_bf16"
152
  if m == 1:
153
+ if k % 32:
154
+ raise RuntimeError("SM120 requires k divisible by 32")
155
  if variant == 4:
156
  return "gemv_fp8_m1_w4"
157
  if variant == 8:
 
167
  return "gemv_fp8_m1_w16"
168
  if variant != 0:
169
  raise RuntimeError("small-M dispatcher currently supports variant=0 only")
170
+ if k % 32:
171
+ raise RuntimeError("SM120 requires k divisible by 32")
172
  if m <= 16:
173
  if k % 256 == 0:
174
  return "ld_fp8_gemm_16x128x256_w4" if n % 128 == 0 else "ld_fp8_gemm_16x64x256_w4"
 
207
 
208
  ``input`` and ``weight`` must be FP8 E4M3 CUDA tensors with shapes
209
  ``(M, K)`` and ``(N, K)``. ``alpha`` is a host float, normally the product
210
+ of static per-tensor input and weight scales. SM110 uses the production
211
+ CUTLASS Sq/T1/Wide dispatcher over full model row counts; SM120 uses the
212
+ hand-tuned M<=64 path.
213
  """
214
 
215
  if out is None:
 
235
  return residual
236
 
237
 
238
+ def fp8_linear_bias_bf16(
239
+ input: torch.Tensor,
240
+ weight: torch.Tensor,
241
+ bias: torch.Tensor,
242
+ alpha: float = 1.0,
243
+ out: torch.Tensor | None = None,
244
+ ) -> torch.Tensor:
245
+ """SM110 FP8 linear with fused BF16 bias and BF16 output."""
246
+ if out is None:
247
+ out = torch.empty(
248
+ (input.shape[0], weight.shape[0]),
249
+ device=input.device,
250
+ dtype=torch.bfloat16,
251
+ )
252
+ ops.fp8_linear_bias_bf16(input, weight, bias, float(alpha), out)
253
+ return out
254
+
255
+
256
+ def fp8_linear_bias_residual_bf16(
257
+ input: torch.Tensor,
258
+ weight: torch.Tensor,
259
+ bias: torch.Tensor,
260
+ residual: torch.Tensor,
261
+ alpha: float = 1.0,
262
+ ) -> torch.Tensor:
263
+ """SM110 fused ``residual += alpha * input @ weight.T + bias``."""
264
+ ops.fp8_linear_bias_residual_bf16(
265
+ input, weight, bias, float(alpha), residual
266
+ )
267
+ return residual
268
+
269
+
270
+ def fp8_linear_bias_gelu_bf16(
271
+ input: torch.Tensor,
272
+ weight: torch.Tensor,
273
+ bias: torch.Tensor,
274
+ alpha: float = 1.0,
275
+ out: torch.Tensor | None = None,
276
+ ) -> torch.Tensor:
277
+ """SM110 FP8 linear with fused BF16 bias and GELU epilogue."""
278
+ if out is None:
279
+ out = torch.empty(
280
+ (input.shape[0], weight.shape[0]),
281
+ device=input.device,
282
+ dtype=torch.bfloat16,
283
+ )
284
+ ops.fp8_linear_bias_gelu_bf16(input, weight, bias, float(alpha), out)
285
+ return out
286
+
287
+
288
  def fp8_blockwise_linear_bf16(
289
  input: torch.Tensor,
290
  weight: torch.Tensor,
 
292
  weight_scale: torch.Tensor,
293
  out: torch.Tensor | None = None,
294
  ) -> torch.Tensor:
295
+ """Block-128 scaled FP8 linear with BF16 output on SM89/SM120."""
296
 
297
  if out is None:
298
  out = torch.empty(
 
336
  __all__ = [
337
  "fp8_linear_bf16",
338
  "fp8_linear_residual_bf16",
339
+ "fp8_linear_bias_bf16",
340
+ "fp8_linear_bias_residual_bf16",
341
+ "fp8_linear_bias_gelu_bf16",
342
  "fp8_blockwise_linear_bf16",
343
  "fp8_blockwise_swiglu_quantize_fp8",
344
  "select_fp8_linear_tile",
build/torch211-cxx11-cu130-x86_64-linux/{_fp8_gemm_cuda_bcb0782.abi3.so → _fp8_gemm_cuda_a6f59d3.abi3.so} RENAMED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:4f556905c27b2b6a3e59bf294b61e699196c14a3b10aa7db3fb0f09a3b709fa4
3
- size 4147648
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:358d4f9503b8f87764588b95f54caed3325b9e10f81d7632890e34f5a7f0bd03
3
+ size 5654672
build/torch211-cxx11-cu130-x86_64-linux/_ops.py CHANGED
@@ -1,9 +1,9 @@
1
  import torch
2
- from . import _fp8_gemm_cuda_bcb0782
3
- ops = torch.ops._fp8_gemm_cuda_bcb0782
4
 
5
  def add_op_namespace_prefix(op_name: str):
6
  """
7
  Prefix op by namespace.
8
  """
9
- return f"_fp8_gemm_cuda_bcb0782::{op_name}"
 
1
  import torch
2
+ from . import _fp8_gemm_cuda_a6f59d3
3
+ ops = torch.ops._fp8_gemm_cuda_a6f59d3
4
 
5
  def add_op_namespace_prefix(op_name: str):
6
  """
7
  Prefix op by namespace.
8
  """
9
+ return f"_fp8_gemm_cuda_a6f59d3::{op_name}"
build/torch211-cxx11-cu130-x86_64-linux/metadata.json CHANGED
@@ -1,12 +1,13 @@
1
  {
2
  "name": "fp8-gemm",
3
- "id": "_fp8_gemm_cuda_bcb0782",
4
  "version": 1,
5
  "license": "Apache-2.0",
6
  "python-depends": [],
7
  "backend": {
8
  "type": "cuda",
9
  "archs": [
 
10
  "12.0a",
11
  "8.9"
12
  ]
@@ -14,9 +15,9 @@
14
  "digest": {
15
  "algorithm": "sha256",
16
  "files": {
17
- "__init__.py": "AhlkcnyTuk3G4aYsvMqu3QdiEZSTA6Y00poqqiX+Mps=",
18
- "_fp8_gemm_cuda_bcb0782.abi3.so": "T1VpBcJ7K2o+Wb8pS2HmmRlsFKOxCqfbP7Dwmjtwn6Q=",
19
- "_ops.py": "qpBYMOKsfVJtHhKzoX/hxuQPF3MGISv+MQwpPmbzUcw=",
20
  "fp8_gemm/__init__.py": "DFYPlrhXwYjEqCl/8n0SmWGZV8NFml5DPhMjKfv98GY="
21
  }
22
  }
 
1
  {
2
  "name": "fp8-gemm",
3
+ "id": "_fp8_gemm_cuda_a6f59d3",
4
  "version": 1,
5
  "license": "Apache-2.0",
6
  "python-depends": [],
7
  "backend": {
8
  "type": "cuda",
9
  "archs": [
10
+ "11.0a",
11
  "12.0a",
12
  "8.9"
13
  ]
 
15
  "digest": {
16
  "algorithm": "sha256",
17
  "files": {
18
+ "__init__.py": "HsFfsGY7AK/6Fziv4r2VQ+R2kDjS6hxKKgWqlA+zrKY=",
19
+ "_fp8_gemm_cuda_a6f59d3.abi3.so": "NY1PlQO4+HdkWIuV9Uyu0zJbnhD4HXYyiQ409afwvQM=",
20
+ "_ops.py": "kgQ6LWi/1cNaF3rEiBHA8t3jyz0toqRVj94wVDPiqVU=",
21
  "fp8_gemm/__init__.py": "DFYPlrhXwYjEqCl/8n0SmWGZV8NFml5DPhMjKfv98GY="
22
  }
23
  }