liangsu9988 commited on
Commit
b699603
·
verified ·
1 Parent(s): 3b83f22

Promote latest kernel artifacts to main

Browse files
README.md CHANGED
@@ -1,9 +1,116 @@
1
- # flashrt/fp4-gemm
2
 
3
- This repository is a compatibility mirror for older `kernels` clients
4
- that resolve repositories through the default Hugging Face model repo API.
5
 
6
- Canonical Kernel Hub repo: https://huggingface.co/kernels/flashrt/fp4-gemm
 
 
 
7
 
8
- Do not edit this mirror by hand. It is generated from the Kernel Hub
9
- `vN` branches and contains the same `build/**` artifacts.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # fp4-gemm
2
 
3
+ FlashRT native Blackwell NVFP4 A4W4 GEMM kernels.
 
4
 
5
+ This package consumes packed FP4 E2M1 tensors plus CUTLASS Sm1xx SFA/SFB scale
6
+ buffers and produces BF16 output. It is designed to pair with
7
+ `flashrt/fp4-fused-ops` and other static low-bit transformer/diffuser runtime
8
+ paths.
9
 
10
+ ## Available Functions
11
+
12
+ - `sfa_size_bytes(rows, dim)`
13
+ - `quantize_fp4_sfa_fp16(x, packed=None, sfa=None, is_sfb=False)`
14
+ - `quantize_fp4_sfa_bf16(x, packed=None, sfa=None, is_sfb=False)`
15
+ - `dequantize_fp4_sfa_fp16(packed, sfa, out=None, is_sfb=False)`
16
+ - `nvfp4_gemm_bf16(a_packed, b_packed, sfa, sfb, alpha=1.0, out=None, variant=-1)`
17
+ - `nvfp4_gemm_bias_bf16(a_packed, b_packed, sfa, sfb, bias, out=None)`
18
+ - `nvfp4_gemm_bias_residual_bf16(a_packed, b_packed, sfa, sfb, bias, residual, out=None)`
19
+ - `nvfp4_gemm_residual_bf16(a_packed, b_packed, sfa, sfb, residual, alpha=1.0, out=None)`
20
+ - `nvfp4_gemm_bias_gelu_bf16(a_packed, b_packed, sfa, sfb, bias, alpha=1.0, out=None)`
21
+ - `nvfp4_gemm_bias_gelu_nvfp4(a_packed, b_packed, sfa, sfb, bias, alpha=1.0, out_packed=None, out_sfa=None)`
22
+ - `nvfp4_gemm_streamk_bf16(a_packed, b_packed, sfa, sfb, alpha=1.0, out=None)`
23
+ - `nvfp4_gemm_streamk_bias_bf16(a_packed, b_packed, sfa, sfb, bias, alpha=1.0, out=None)`
24
+ - `fp4_w4a16_linear_bf16(...)` is retained as a compatibility alias
25
+
26
+ ## Tensor Contract
27
+
28
+ - `a_packed`: `torch.uint8`, shape `(M, K / 2)`.
29
+ - `b_packed`: `torch.uint8`, shape `(N, K / 2)`.
30
+ - `sfa`: `torch.uint8`, CUTLASS SFA layout for `(M, K)`.
31
+ - `sfb`: `torch.uint8`, CUTLASS SFB layout for `(N, K)`.
32
+ - output: `torch.bfloat16`, shape `(M, N)`.
33
+ - `K` must be divisible by 16.
34
+ - Targets: Blackwell `sm_110a` (Jetson AGX Thor, CUDA 13+) and `sm_120a`
35
+ (RTX Blackwell, CUDA 12.8+).
36
+
37
+ `variant` selects the CUTLASS schedule:
38
+
39
+ - `-1`: architecture-aware auto-dispatch (public default).
40
+ - `0`: default `<128,128,256>` cooperative schedule.
41
+ - `1`: widen `<128,256,128>` schedule, intended for very large `N`.
42
+ - `2`: pingpong schedule for A/B testing shape-specific wins.
43
+
44
+ The canonical linear API and FP4/SFA quantize/dequantize helpers are available
45
+ on both SM110 and SM120. SM110 additionally provides the GROOT N1.7 production
46
+ epilogues `nvfp4_gemm_bias_bf16`, `nvfp4_gemm_bias_residual_bf16`, and
47
+ `nvfp4_gemm_bias_gelu_nvfp4`. The latter emits packed FP4 plus CUTLASS SFA so
48
+ the following projection can consume it without a BF16 materialization and a
49
+ standalone quantization launch. Stream-K and the older BF16 GELU epilogue keep
50
+ their existing SM120 dispatch and reject unsupported architectures explicitly.
51
+
52
+ The SM110 release gate includes the production `(M,N,K)` shapes
53
+ `(41,4608,1536)`, `(41,6144,1536)`, and `(41,1536,6144)`, plus the legacy
54
+ `M=51` compatibility row. The kernels are the native sources used by FlashRT's
55
+ GROOT N1.7 Thor NVFP4 pipeline.
56
+
57
+ ## Minimal Usage
58
+
59
+ ```python
60
+ from kernels import get_kernel
61
+ import torch
62
+
63
+ ops = get_kernel("flashrt/fp4-gemm", version=1, trust_remote_code=True)
64
+
65
+ x = torch.randn((32, 256), device="cuda", dtype=torch.float16)
66
+ w = torch.randn((512, 256), device="cuda", dtype=torch.float16)
67
+
68
+ a_packed, sfa = ops.quantize_fp4_sfa_fp16(x, is_sfb=False)
69
+ b_packed, sfb = ops.quantize_fp4_sfa_fp16(w, is_sfb=True)
70
+
71
+ y = ops.nvfp4_gemm_bf16(a_packed, b_packed, sfa, sfb, alpha=1.0)
72
+ ```
73
+
74
+ For BF16 model activations, use the direct producer so the hot path does not
75
+ materialize an intermediate FP16 tensor:
76
+
77
+ ```python
78
+ x_bf16 = torch.randn((1, 5120), device="cuda", dtype=torch.bfloat16)
79
+ a_packed, sfa = ops.quantize_fp4_sfa_bf16(x_bf16)
80
+ ```
81
+
82
+ The BF16 entry writes the same E2M1 bytes and CUTLASS SFA/SFB layout as
83
+ `quantize_fp4_sfa_fp16(x_bf16.to(torch.float16))` for finite FP16-range
84
+ inputs. It is an additive API; the existing FP16 producer remains unchanged.
85
+
86
+ The quantize/dequantize helpers are included for examples and validation. A
87
+ production runtime should keep weights prepacked and should avoid quantizing in
88
+ the hot path unless that producer kernel is part of the intended low-bit block.
89
+
90
+ Use the bias/GELU and residual variants to avoid returning to BF16
91
+ elementwise code between low-bit GEMMs. Stream-K variants are selected only
92
+ for the validated large down-projection shapes; unsupported shapes reject
93
+ rather than silently selecting a losing schedule.
94
+
95
+ ## Validation
96
+
97
+ ```bash
98
+ python fp4-gemm/tests/test_fp4_gemm.py --backend source --mode full
99
+ python fp4-gemm/tests/test_fp4_gemm.py --backend installed --mode full \
100
+ --artifact fp4-gemm/build/torch211-cxx11-cu128-x86_64-linux
101
+ python fp4-gemm/benchmarks/benchmark.py --backend installed --mode headline \
102
+ --artifact fp4-gemm/build/torch211-cxx11-cu128-x86_64-linux
103
+
104
+ # Thor model-shape gate
105
+ python fp4-gemm/tests/test_fp4_gemm.py --backend installed \
106
+ --mode thor-models \
107
+ --artifact fp4-gemm/build/torch211-cxx11-cu130-aarch64-linux
108
+ ```
109
+
110
+ The correctness reference dequantizes the same FP4/SFA and FP4/SFB inputs used
111
+ by the kernel, then computes the PyTorch GEMM reference from those dequantized
112
+ low-bit values.
113
+
114
+ The producer gate also checks the BF16 direct entry byte-for-byte against the
115
+ established FP16 compatibility chain at decode widths 5120, 6144 and 17408,
116
+ plus multi-row activation and SFB layouts.
build/torch213-cxx11-cu130-aarch64-linux/__init__.py ADDED
@@ -0,0 +1,358 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """FlashRT FP4 GEMM kernels."""
2
+
3
+ from __future__ import annotations
4
+
5
+ import torch
6
+
7
+ from ._ops import add_op_namespace_prefix, ops
8
+
9
+
10
+ def sfa_size_bytes(rows: int, dim: int) -> int:
11
+ if rows <= 0 or dim <= 0 or dim % 16 != 0:
12
+ raise ValueError("rows must be positive and dim must be positive/divisible by 16")
13
+ n_blocks = dim // 16
14
+ n_row_super = (rows + 127) // 128
15
+ n_col_super = (n_blocks + 3) // 4
16
+ return n_row_super * n_col_super * 512
17
+
18
+
19
+ def _alloc_fp4(rows: int, dim: int, device: torch.device | str):
20
+ return (
21
+ torch.empty((rows, dim // 2), device=device, dtype=torch.uint8),
22
+ torch.empty((sfa_size_bytes(rows, dim),), device=device, dtype=torch.uint8),
23
+ )
24
+
25
+
26
+ @torch.library.register_fake(add_op_namespace_prefix("nvfp4_gemm_bf16"))
27
+ def _linear_fake(
28
+ a_packed: torch.Tensor,
29
+ b_packed: torch.Tensor,
30
+ sfa: torch.Tensor,
31
+ sfb: torch.Tensor,
32
+ out: torch.Tensor,
33
+ alpha: float = 1.0,
34
+ variant: int = -1,
35
+ ) -> None:
36
+ return None
37
+
38
+
39
+ @torch.library.register_fake(add_op_namespace_prefix("fp4_w4a4_gemv_warpsplit_bf16"))
40
+ def _gemv_warpsplit_fake(a_packed, b_packed, sfa, sfb, out, alpha: float = 1.0, warps: int = 4, stages: int = 4) -> None:
41
+ if a_packed.shape[0] != 1:
42
+ raise RuntimeError("warp-split GEMV serves M=1 only")
43
+ if out.shape != (1, b_packed.shape[0]):
44
+ raise RuntimeError("out must have shape (1, N)")
45
+ return None
46
+
47
+
48
+ @torch.library.register_fake(add_op_namespace_prefix("fp4_w4a16_linear_bf16"))
49
+ def _legacy_linear_fake(
50
+ a_packed: torch.Tensor,
51
+ b_packed: torch.Tensor,
52
+ sfa: torch.Tensor,
53
+ sfb: torch.Tensor,
54
+ out: torch.Tensor,
55
+ alpha: float = 1.0,
56
+ variant: int = -1,
57
+ ) -> None:
58
+ return None
59
+
60
+
61
+ @torch.library.register_fake(add_op_namespace_prefix("nvfp4_gemm_bias_bf16"))
62
+ def _bias_fake(a, b, sfa, sfb, bias, out) -> None:
63
+ return None
64
+
65
+
66
+ @torch.library.register_fake(add_op_namespace_prefix("nvfp4_gemm_bias_residual_bf16"))
67
+ def _bias_residual_fake(a, b, sfa, sfb, bias, residual, out) -> None:
68
+ return None
69
+
70
+
71
+ @torch.library.register_fake(add_op_namespace_prefix("quantize_fp4_sfa_fp16"))
72
+ def _quant_fake(x: torch.Tensor, packed: torch.Tensor, sfa: torch.Tensor, is_sfb: bool = False) -> None:
73
+ return None
74
+
75
+
76
+ @torch.library.register_fake(add_op_namespace_prefix("quantize_fp4_sfa_bf16"))
77
+ def _quant_bf16_fake(x: torch.Tensor, packed: torch.Tensor, sfa: torch.Tensor, is_sfb: bool = False) -> None:
78
+ return None
79
+
80
+
81
+ @torch.library.register_fake(add_op_namespace_prefix("dequantize_fp4_sfa_fp16"))
82
+ def _dequant_fake(packed: torch.Tensor, sfa: torch.Tensor, out: torch.Tensor, is_sfb: bool = False) -> None:
83
+ return None
84
+
85
+
86
+ @torch.library.register_fake(add_op_namespace_prefix("nvfp4_gemm_residual_bf16"))
87
+ def _residual_fake(a, b, sfa, sfb, residual, out, alpha: float = 1.0) -> None:
88
+ return None
89
+
90
+
91
+ @torch.library.register_fake(add_op_namespace_prefix("nvfp4_gemm_bias_gelu_bf16"))
92
+ def _bias_gelu_fake(a, b, sfa, sfb, bias, out, alpha: float = 1.0) -> None:
93
+ return None
94
+
95
+
96
+ @torch.library.register_fake(add_op_namespace_prefix("nvfp4_gemm_bias_gelu_nvfp4"))
97
+ def _bias_gelu_nvfp4_fake(
98
+ a, b, sfa, sfb, bias, out_packed, out_sfa, alpha: float = 1.0
99
+ ) -> None:
100
+ return None
101
+
102
+
103
+ @torch.library.register_fake(add_op_namespace_prefix("nvfp4_gemm_streamk_bf16"))
104
+ def _streamk_fake(a, b, sfa, sfb, out, alpha: float = 1.0) -> None:
105
+ return None
106
+
107
+
108
+ @torch.library.register_fake(add_op_namespace_prefix("nvfp4_gemm_streamk_bias_bf16"))
109
+ def _streamk_bias_fake(a, b, sfa, sfb, bias, out, alpha: float = 1.0) -> None:
110
+ return None
111
+
112
+
113
+ def quantize_fp4_sfa_fp16(
114
+ x: torch.Tensor,
115
+ packed: torch.Tensor | None = None,
116
+ sfa: torch.Tensor | None = None,
117
+ is_sfb: bool = False,
118
+ ):
119
+ if packed is None or sfa is None:
120
+ packed, sfa = _alloc_fp4(x.shape[0], x.shape[1], x.device)
121
+ ops.quantize_fp4_sfa_fp16(x, packed, sfa, bool(is_sfb))
122
+ return packed, sfa
123
+
124
+
125
+ def quantize_fp4_sfa_bf16(
126
+ x: torch.Tensor,
127
+ packed: torch.Tensor | None = None,
128
+ sfa: torch.Tensor | None = None,
129
+ is_sfb: bool = False,
130
+ ):
131
+ """Quantize BF16 directly to packed E2M1 and CUTLASS SFA/SFB."""
132
+ if packed is None or sfa is None:
133
+ packed, sfa = _alloc_fp4(x.shape[0], x.shape[1], x.device)
134
+ ops.quantize_fp4_sfa_bf16(x, packed, sfa, bool(is_sfb))
135
+ return packed, sfa
136
+
137
+
138
+ def dequantize_fp4_sfa_fp16(
139
+ packed: torch.Tensor,
140
+ sfa: torch.Tensor,
141
+ out: torch.Tensor | None = None,
142
+ is_sfb: bool = False,
143
+ ) -> torch.Tensor:
144
+ if out is None:
145
+ out = torch.empty((packed.shape[0], packed.shape[1] * 2), device=packed.device, dtype=torch.float16)
146
+ ops.dequantize_fp4_sfa_fp16(packed, sfa, out, bool(is_sfb))
147
+ return out
148
+
149
+
150
+ def nvfp4_gemm_bf16(
151
+ a_packed: torch.Tensor,
152
+ b_packed: torch.Tensor,
153
+ sfa: torch.Tensor,
154
+ sfb: torch.Tensor,
155
+ alpha: float = 1.0,
156
+ out: torch.Tensor | None = None,
157
+ variant: int = -1,
158
+ ) -> torch.Tensor:
159
+ if out is None:
160
+ out = torch.empty((a_packed.shape[0], b_packed.shape[0]), device=a_packed.device, dtype=torch.bfloat16)
161
+ ops.nvfp4_gemm_bf16(a_packed, b_packed, sfa, sfb, out, float(alpha), int(variant))
162
+ return out
163
+
164
+
165
+ def fp4_w4a4_gemv_warpsplit_bf16(
166
+ a_packed: torch.Tensor,
167
+ b_packed: torch.Tensor,
168
+ sfa: torch.Tensor,
169
+ sfb: torch.Tensor,
170
+ *,
171
+ alpha: float = 1.0,
172
+ warps: int = 4,
173
+ stages: int = 4,
174
+ out: Optional[torch.Tensor] = None,
175
+ ) -> torch.Tensor:
176
+ """Warp-split-K NVFP4 W4A4 GEMV for the M=1 decode row (SM120).
177
+
178
+ Splits K across warps inside one block with a shared-memory reduce -
179
+ no cross-block intermediate, so it stays safe under CUDA-graph
180
+ replay - and fills the SMs the tiled GEMM underfills at long-K
181
+ small-M decode shapes. Same packed/scale layouts as the linear
182
+ entry points."""
183
+ if out is None:
184
+ out = torch.empty((1, b_packed.shape[0]), device=a_packed.device, dtype=torch.bfloat16)
185
+ ops.fp4_w4a4_gemv_warpsplit_bf16(a_packed, b_packed, sfa, sfb, out, float(alpha), int(warps), int(stages))
186
+ return out
187
+
188
+
189
+ def fp4_w4a16_linear_bf16(
190
+ a_packed: torch.Tensor,
191
+ b_packed: torch.Tensor,
192
+ sfa: torch.Tensor,
193
+ sfb: torch.Tensor,
194
+ alpha: float = 1.0,
195
+ out: torch.Tensor | None = None,
196
+ variant: int = -1,
197
+ ) -> torch.Tensor:
198
+ """Compatibility alias for :func:`nvfp4_gemm_bf16`."""
199
+ return nvfp4_gemm_bf16(
200
+ a_packed, b_packed, sfa, sfb, alpha=alpha, out=out, variant=variant
201
+ )
202
+
203
+
204
+ def nvfp4_gemm_bias_bf16(
205
+ a_packed: torch.Tensor,
206
+ b_packed: torch.Tensor,
207
+ sfa: torch.Tensor,
208
+ sfb: torch.Tensor,
209
+ bias: torch.Tensor,
210
+ *,
211
+ out: torch.Tensor | None = None,
212
+ ) -> torch.Tensor:
213
+ """SM110 NVFP4 GEMM with a fused per-column BF16 bias."""
214
+ if out is None:
215
+ out = torch.empty(
216
+ (a_packed.shape[0], b_packed.shape[0]),
217
+ device=a_packed.device,
218
+ dtype=torch.bfloat16,
219
+ )
220
+ ops.nvfp4_gemm_bias_bf16(a_packed, b_packed, sfa, sfb, bias, out)
221
+ return out
222
+
223
+
224
+ def nvfp4_gemm_bias_residual_bf16(
225
+ a_packed: torch.Tensor,
226
+ b_packed: torch.Tensor,
227
+ sfa: torch.Tensor,
228
+ sfb: torch.Tensor,
229
+ bias: torch.Tensor,
230
+ residual: torch.Tensor,
231
+ *,
232
+ out: torch.Tensor | None = None,
233
+ ) -> torch.Tensor:
234
+ """SM110 NVFP4 GEMM with fused BF16 bias and residual add."""
235
+ if out is None:
236
+ out = torch.empty_like(residual)
237
+ ops.nvfp4_gemm_bias_residual_bf16(
238
+ a_packed, b_packed, sfa, sfb, bias, residual, out
239
+ )
240
+ return out
241
+
242
+
243
+ def nvfp4_gemm_residual_bf16(
244
+ a_packed: torch.Tensor,
245
+ b_packed: torch.Tensor,
246
+ sfa: torch.Tensor,
247
+ sfb: torch.Tensor,
248
+ residual: torch.Tensor,
249
+ alpha: float = 1.0,
250
+ out: torch.Tensor | None = None,
251
+ ) -> torch.Tensor:
252
+ if out is None:
253
+ out = torch.empty_like(residual)
254
+ ops.nvfp4_gemm_residual_bf16(
255
+ a_packed, b_packed, sfa, sfb, residual, out, float(alpha)
256
+ )
257
+ return out
258
+
259
+
260
+ def nvfp4_gemm_bias_gelu_bf16(
261
+ a_packed: torch.Tensor,
262
+ b_packed: torch.Tensor,
263
+ sfa: torch.Tensor,
264
+ sfb: torch.Tensor,
265
+ bias: torch.Tensor,
266
+ alpha: float = 1.0,
267
+ out: torch.Tensor | None = None,
268
+ ) -> torch.Tensor:
269
+ if out is None:
270
+ out = torch.empty(
271
+ (a_packed.shape[0], b_packed.shape[0]),
272
+ device=a_packed.device,
273
+ dtype=torch.bfloat16,
274
+ )
275
+ ops.nvfp4_gemm_bias_gelu_bf16(
276
+ a_packed, b_packed, sfa, sfb, bias, out, float(alpha)
277
+ )
278
+ return out
279
+
280
+
281
+ def nvfp4_gemm_bias_gelu_nvfp4(
282
+ a_packed: torch.Tensor,
283
+ b_packed: torch.Tensor,
284
+ sfa: torch.Tensor,
285
+ sfb: torch.Tensor,
286
+ bias: torch.Tensor,
287
+ alpha: float = 1.0,
288
+ out_packed: torch.Tensor | None = None,
289
+ out_sfa: torch.Tensor | None = None,
290
+ ) -> tuple[torch.Tensor, torch.Tensor]:
291
+ m, n = a_packed.shape[0], b_packed.shape[0]
292
+ if out_packed is None:
293
+ out_packed = torch.empty((m, n // 2), device=a_packed.device, dtype=torch.uint8)
294
+ if out_sfa is None:
295
+ out_sfa = torch.empty((sfa_size_bytes(m, n),), device=a_packed.device, dtype=torch.uint8)
296
+ ops.nvfp4_gemm_bias_gelu_nvfp4(
297
+ a_packed, b_packed, sfa, sfb, bias, out_packed, out_sfa, float(alpha)
298
+ )
299
+ return out_packed, out_sfa
300
+
301
+
302
+ def nvfp4_gemm_streamk_bf16(
303
+ a_packed: torch.Tensor,
304
+ b_packed: torch.Tensor,
305
+ sfa: torch.Tensor,
306
+ sfb: torch.Tensor,
307
+ alpha: float = 1.0,
308
+ out: torch.Tensor | None = None,
309
+ ) -> torch.Tensor:
310
+ if out is None:
311
+ out = torch.empty(
312
+ (a_packed.shape[0], b_packed.shape[0]),
313
+ device=a_packed.device,
314
+ dtype=torch.bfloat16,
315
+ )
316
+ ops.nvfp4_gemm_streamk_bf16(
317
+ a_packed, b_packed, sfa, sfb, out, float(alpha)
318
+ )
319
+ return out
320
+
321
+
322
+ def nvfp4_gemm_streamk_bias_bf16(
323
+ a_packed: torch.Tensor,
324
+ b_packed: torch.Tensor,
325
+ sfa: torch.Tensor,
326
+ sfb: torch.Tensor,
327
+ bias: torch.Tensor,
328
+ alpha: float = 1.0,
329
+ out: torch.Tensor | None = None,
330
+ ) -> torch.Tensor:
331
+ if out is None:
332
+ out = torch.empty(
333
+ (a_packed.shape[0], b_packed.shape[0]),
334
+ device=a_packed.device,
335
+ dtype=torch.bfloat16,
336
+ )
337
+ ops.nvfp4_gemm_streamk_bias_bf16(
338
+ a_packed, b_packed, sfa, sfb, bias, out, float(alpha)
339
+ )
340
+ return out
341
+
342
+
343
+ __all__ = [
344
+ "dequantize_fp4_sfa_fp16",
345
+ "fp4_w4a16_linear_bf16",
346
+ "fp4_w4a4_gemv_warpsplit_bf16",
347
+ "nvfp4_gemm_bf16",
348
+ "nvfp4_gemm_bias_bf16",
349
+ "nvfp4_gemm_bias_gelu_bf16",
350
+ "nvfp4_gemm_bias_gelu_nvfp4",
351
+ "nvfp4_gemm_bias_residual_bf16",
352
+ "nvfp4_gemm_residual_bf16",
353
+ "nvfp4_gemm_streamk_bf16",
354
+ "nvfp4_gemm_streamk_bias_bf16",
355
+ "quantize_fp4_sfa_fp16",
356
+ "quantize_fp4_sfa_bf16",
357
+ "sfa_size_bytes",
358
+ ]
build/torch213-cxx11-cu130-aarch64-linux/_fp4_gemm_cuda_8a66d8b.abi3.so ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:35bd2043a1a1880f362954fe72a22651017d30a585d38ed5d2ffe86de286b45a
3
+ size 1118808
build/torch213-cxx11-cu130-aarch64-linux/_ops.py ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ import torch
2
+ from . import _fp4_gemm_cuda_8a66d8b
3
+ ops = torch.ops._fp4_gemm_cuda_8a66d8b
4
+
5
+ def add_op_namespace_prefix(op_name: str):
6
+ return f"_fp4_gemm_cuda_8a66d8b::{op_name}"
build/torch213-cxx11-cu130-aarch64-linux/fp4_gemm/__init__.py ADDED
@@ -0,0 +1,14 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import ctypes
2
+ import importlib.util
3
+ import sys
4
+ from pathlib import Path
5
+
6
+ def _import_from_path(file_path: Path):
7
+ path_hash = '{:x}'.format(ctypes.c_size_t(hash(file_path.absolute())).value)
8
+ spec = importlib.util.spec_from_file_location(path_hash, file_path)
9
+ module = importlib.util.module_from_spec(spec)
10
+ sys.modules[path_hash] = module
11
+ spec.loader.exec_module(module)
12
+ return module
13
+
14
+ globals().update(vars(_import_from_path(Path(__file__).parent.parent / '__init__.py')))
build/torch213-cxx11-cu130-aarch64-linux/metadata.json ADDED
@@ -0,0 +1,28 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "name": "fp4-gemm",
3
+ "id": "_fp4_gemm_cuda_8a66d8b",
4
+ "version": 1,
5
+ "license": "Apache-2.0",
6
+ "python-depends": [],
7
+ "backend": {
8
+ "type": "cuda",
9
+ "archs": [
10
+ "11.0a"
11
+ ]
12
+ },
13
+ "digest": {
14
+ "algorithm": "sha256",
15
+ "files": {
16
+ "__init__.py": "+Kk/VNnIWwe9nszQYvL2qtU3DpKxbqHVfklg5WtVuB4=",
17
+ "_fp4_gemm_cuda_8a66d8b.abi3.so": "Nb0gQ6GhiA82KVT+cqImUQF9MKWF047V0v/obeKGtFo=",
18
+ "_ops.py": "/hna1MwmSjR4pewWf12X5smsDRcaiHuoR2Gri/PrBFc=",
19
+ "fp4_gemm/__init__.py": "v6p5XMfQzddhi1fLSAw4HX9CyS0rQsidvu9VsT01xi4="
20
+ }
21
+ },
22
+ "provenance": {
23
+ "kernel": {
24
+ "sha": "8a66d8b",
25
+ "dirty": false
26
+ }
27
+ }
28
+ }