liangsu9988 commited on
Commit
52eec20
·
verified ·
1 Parent(s): 271000a

Promote latest kernel artifacts to main

Browse files
README.md CHANGED
@@ -1,9 +1,116 @@
1
- # flashrt/fp4-gemm
2
 
3
- This repository is a compatibility mirror for older `kernels` clients
4
- that resolve repositories through the default Hugging Face model repo API.
5
 
6
- Canonical Kernel Hub repo: https://huggingface.co/kernels/flashrt/fp4-gemm
 
 
 
7
 
8
- Do not edit this mirror by hand. It is generated from the Kernel Hub
9
- `vN` branches and contains the same `build/**` artifacts.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # fp4-gemm
2
 
3
+ FlashRT native Blackwell NVFP4 A4W4 GEMM kernels.
 
4
 
5
+ This package consumes packed FP4 E2M1 tensors plus CUTLASS Sm1xx SFA/SFB scale
6
+ buffers and produces BF16 output. It is designed to pair with
7
+ `flashrt/fp4-fused-ops` and other static low-bit transformer/diffuser runtime
8
+ paths.
9
 
10
+ ## Available Functions
11
+
12
+ - `sfa_size_bytes(rows, dim)`
13
+ - `quantize_fp4_sfa_fp16(x, packed=None, sfa=None, is_sfb=False)`
14
+ - `quantize_fp4_sfa_bf16(x, packed=None, sfa=None, is_sfb=False)`
15
+ - `dequantize_fp4_sfa_fp16(packed, sfa, out=None, is_sfb=False)`
16
+ - `nvfp4_gemm_bf16(a_packed, b_packed, sfa, sfb, alpha=1.0, out=None, variant=-1)`
17
+ - `nvfp4_gemm_bias_bf16(a_packed, b_packed, sfa, sfb, bias, out=None)`
18
+ - `nvfp4_gemm_bias_residual_bf16(a_packed, b_packed, sfa, sfb, bias, residual, out=None)`
19
+ - `nvfp4_gemm_residual_bf16(a_packed, b_packed, sfa, sfb, residual, alpha=1.0, out=None)`
20
+ - `nvfp4_gemm_bias_gelu_bf16(a_packed, b_packed, sfa, sfb, bias, alpha=1.0, out=None)`
21
+ - `nvfp4_gemm_bias_gelu_nvfp4(a_packed, b_packed, sfa, sfb, bias, alpha=1.0, out_packed=None, out_sfa=None)`
22
+ - `nvfp4_gemm_streamk_bf16(a_packed, b_packed, sfa, sfb, alpha=1.0, out=None)`
23
+ - `nvfp4_gemm_streamk_bias_bf16(a_packed, b_packed, sfa, sfb, bias, alpha=1.0, out=None)`
24
+ - `fp4_w4a16_linear_bf16(...)` is retained as a compatibility alias
25
+
26
+ ## Tensor Contract
27
+
28
+ - `a_packed`: `torch.uint8`, shape `(M, K / 2)`.
29
+ - `b_packed`: `torch.uint8`, shape `(N, K / 2)`.
30
+ - `sfa`: `torch.uint8`, CUTLASS SFA layout for `(M, K)`.
31
+ - `sfb`: `torch.uint8`, CUTLASS SFB layout for `(N, K)`.
32
+ - output: `torch.bfloat16`, shape `(M, N)`.
33
+ - `K` must be divisible by 16.
34
+ - Targets: Blackwell `sm_110a` (Jetson AGX Thor, CUDA 13+) and `sm_120a`
35
+ (RTX Blackwell, CUDA 12.8+).
36
+
37
+ `variant` selects the CUTLASS schedule:
38
+
39
+ - `-1`: architecture-aware auto-dispatch (public default).
40
+ - `0`: default `<128,128,256>` cooperative schedule.
41
+ - `1`: widen `<128,256,128>` schedule, intended for very large `N`.
42
+ - `2`: pingpong schedule for A/B testing shape-specific wins.
43
+
44
+ The canonical linear API and FP4/SFA quantize/dequantize helpers are available
45
+ on both SM110 and SM120. SM110 additionally provides the GROOT N1.7 production
46
+ epilogues `nvfp4_gemm_bias_bf16`, `nvfp4_gemm_bias_residual_bf16`, and
47
+ `nvfp4_gemm_bias_gelu_nvfp4`. The latter emits packed FP4 plus CUTLASS SFA so
48
+ the following projection can consume it without a BF16 materialization and a
49
+ standalone quantization launch. Stream-K and the older BF16 GELU epilogue keep
50
+ their existing SM120 dispatch and reject unsupported architectures explicitly.
51
+
52
+ The SM110 release gate includes the production `(M,N,K)` shapes
53
+ `(41,4608,1536)`, `(41,6144,1536)`, and `(41,1536,6144)`, plus the legacy
54
+ `M=51` compatibility row. The kernels are the native sources used by FlashRT's
55
+ GROOT N1.7 Thor NVFP4 pipeline.
56
+
57
+ ## Minimal Usage
58
+
59
+ ```python
60
+ from kernels import get_kernel
61
+ import torch
62
+
63
+ ops = get_kernel("flashrt/fp4-gemm", version=1, trust_remote_code=True)
64
+
65
+ x = torch.randn((32, 256), device="cuda", dtype=torch.float16)
66
+ w = torch.randn((512, 256), device="cuda", dtype=torch.float16)
67
+
68
+ a_packed, sfa = ops.quantize_fp4_sfa_fp16(x, is_sfb=False)
69
+ b_packed, sfb = ops.quantize_fp4_sfa_fp16(w, is_sfb=True)
70
+
71
+ y = ops.nvfp4_gemm_bf16(a_packed, b_packed, sfa, sfb, alpha=1.0)
72
+ ```
73
+
74
+ For BF16 model activations, use the direct producer so the hot path does not
75
+ materialize an intermediate FP16 tensor:
76
+
77
+ ```python
78
+ x_bf16 = torch.randn((1, 5120), device="cuda", dtype=torch.bfloat16)
79
+ a_packed, sfa = ops.quantize_fp4_sfa_bf16(x_bf16)
80
+ ```
81
+
82
+ The BF16 entry writes the same E2M1 bytes and CUTLASS SFA/SFB layout as
83
+ `quantize_fp4_sfa_fp16(x_bf16.to(torch.float16))` for finite FP16-range
84
+ inputs. It is an additive API; the existing FP16 producer remains unchanged.
85
+
86
+ The quantize/dequantize helpers are included for examples and validation. A
87
+ production runtime should keep weights prepacked and should avoid quantizing in
88
+ the hot path unless that producer kernel is part of the intended low-bit block.
89
+
90
+ Use the bias/GELU and residual variants to avoid returning to BF16
91
+ elementwise code between low-bit GEMMs. Stream-K variants are selected only
92
+ for the validated large down-projection shapes; unsupported shapes reject
93
+ rather than silently selecting a losing schedule.
94
+
95
+ ## Validation
96
+
97
+ ```bash
98
+ python fp4-gemm/tests/test_fp4_gemm.py --backend source --mode full
99
+ python fp4-gemm/tests/test_fp4_gemm.py --backend installed --mode full \
100
+ --artifact fp4-gemm/build/torch211-cxx11-cu128-x86_64-linux
101
+ python fp4-gemm/benchmarks/benchmark.py --backend installed --mode headline \
102
+ --artifact fp4-gemm/build/torch211-cxx11-cu128-x86_64-linux
103
+
104
+ # Thor model-shape gate
105
+ python fp4-gemm/tests/test_fp4_gemm.py --backend installed \
106
+ --mode thor-models \
107
+ --artifact fp4-gemm/build/torch211-cxx11-cu130-aarch64-linux
108
+ ```
109
+
110
+ The correctness reference dequantizes the same FP4/SFA and FP4/SFB inputs used
111
+ by the kernel, then computes the PyTorch GEMM reference from those dequantized
112
+ low-bit values.
113
+
114
+ The producer gate also checks the BF16 direct entry byte-for-byte against the
115
+ established FP16 compatibility chain at decode widths 5120, 6144 and 17408,
116
+ plus multi-row activation and SFB layouts.
build/torch211-cxx11-cu128-x86_64-linux/__init__.py CHANGED
@@ -25,6 +25,15 @@ def capabilities() -> dict[str, object]:
25
  "cuda_capabilities": SUPPORTED_CUDA_CAPABILITIES,
26
  "public_m_alignment": 1,
27
  "raw_sm120_tile_m": 128,
 
 
 
 
 
 
 
 
 
28
  "errors": "exceptions",
29
  }
30
 
@@ -98,6 +107,34 @@ def _gemv_warpsplit_fake(a_packed, b_packed, sfa, sfb, out, alpha: float = 1.0,
98
  return None
99
 
100
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
101
  @torch.library.register_fake(add_op_namespace_prefix("fp4_w4a16_linear_bf16"))
102
  def _legacy_linear_fake(
103
  a_packed: torch.Tensor,
@@ -581,6 +618,89 @@ def fp4_w4a4_gemv_warpsplit_bf16(
581
  return out
582
 
583
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
584
  def fp4_w4a16_linear_bf16(
585
  a_packed: torch.Tensor,
586
  b_packed: torch.Tensor,
@@ -747,11 +867,15 @@ __all__ = [
747
  "dequantize_fp4_sfa_fp16",
748
  "e0m3_weight_gemm_fp16",
749
  "fp4_w4a16_linear_bf16",
 
750
  "fp4_w4a4_gemv_warpsplit_bf16",
 
751
  "nvfp4_gemm_bf16",
752
  "nvfp4_gemm_fp16",
753
  "nvfp4_gemm_variant_bf16",
754
  "nvfp4_gemm_nvfp4",
 
 
755
  "nvfp4_gemm_geglu_nvfp4_fp16",
756
  "nvfp4_gemm_bias_gelu_nvfp4_fp16",
757
  "nvfp4_gemm_bias_residual_fp16",
 
25
  "cuda_capabilities": SUPPORTED_CUDA_CAPABILITIES,
26
  "public_m_alignment": 1,
27
  "raw_sm120_tile_m": 128,
28
+ "sm120_m256_min_m": 512,
29
+ "sm120_m256_qualified_nk": (
30
+ (17408, 5120),
31
+ (5120, 17408),
32
+ (12288, 5120),
33
+ ),
34
+ "sm120_m256_diagnostic_nk": ((16384, 5120),),
35
+ "sm120_interleaved_gemv_m": 1,
36
+ "sm120_interleaved_weight_layout": "groups-of-8 x K/64 x 8 x 32B",
37
  "errors": "exceptions",
38
  }
39
 
 
107
  return None
108
 
109
 
110
+ @torch.library.register_fake(add_op_namespace_prefix("fp4_repack_b_interleaved_sm120"))
111
+ def _repack_b_interleaved_fake(b_packed, b_interleaved) -> None:
112
+ if b_interleaved.shape != b_packed.shape:
113
+ raise RuntimeError("b_interleaved must have the same shape as b_packed")
114
+ return None
115
+
116
+
117
+ @torch.library.register_fake(add_op_namespace_prefix("fp4_w4a4_gemv_warpsplit_interleaved_bf16"))
118
+ def _gemv_warpsplit_interleaved_fake(
119
+ a_packed, b_interleaved, sfa, sfb, out,
120
+ alpha: float = 1.0, warps: int = 4, stages: int = 4,
121
+ ) -> None:
122
+ if a_packed.shape[0] != 1:
123
+ raise RuntimeError("interleaved warp-split GEMV serves M=1 only")
124
+ if out.shape != (1, b_interleaved.shape[0]):
125
+ raise RuntimeError("out must have shape (1, N)")
126
+ return None
127
+
128
+
129
+ @torch.library.register_fake(add_op_namespace_prefix("nvfp4_gemm_m256_bf16"))
130
+ def _m256_fake(a_packed, b_packed, sfa, sfb, workspace, out, alpha: float = 1.0) -> None:
131
+ if a_packed.shape[0] < 512:
132
+ raise RuntimeError("the M256 tier requires M >= 512")
133
+ if out.shape != (a_packed.shape[0], b_packed.shape[0]):
134
+ raise RuntimeError("out must have shape (M, N)")
135
+ return None
136
+
137
+
138
  @torch.library.register_fake(add_op_namespace_prefix("fp4_w4a16_linear_bf16"))
139
  def _legacy_linear_fake(
140
  a_packed: torch.Tensor,
 
618
  return out
619
 
620
 
621
+ def fp4_repack_b_interleaved_sm120(
622
+ b_packed: torch.Tensor,
623
+ *,
624
+ out: Optional[torch.Tensor] = None,
625
+ ) -> torch.Tensor:
626
+ """Repack dense row-major packed FP4 weights for the SM120 M=1 GEMV.
627
+
628
+ This is a bind-time operation. Cache the returned tensor with the packed
629
+ weight; do not execute it in the decode hot path.
630
+ """
631
+ if out is None:
632
+ out = torch.empty_like(b_packed)
633
+ ops.fp4_repack_b_interleaved_sm120(b_packed, out)
634
+ return out
635
+
636
+
637
+ def fp4_w4a4_gemv_warpsplit_interleaved_bf16(
638
+ a_packed: torch.Tensor,
639
+ b_interleaved: torch.Tensor,
640
+ sfa: torch.Tensor,
641
+ sfb: torch.Tensor,
642
+ *,
643
+ alpha: float = 1.0,
644
+ warps: int = 8,
645
+ stages: int = 3,
646
+ out: Optional[torch.Tensor] = None,
647
+ ) -> torch.Tensor:
648
+ """M=1 SM120 W4A4 GEMV using bind-time interleaved FP4 weights."""
649
+ if out is None:
650
+ out = torch.empty(
651
+ (1, b_interleaved.shape[0]),
652
+ device=a_packed.device,
653
+ dtype=torch.bfloat16,
654
+ )
655
+ ops.fp4_w4a4_gemv_warpsplit_interleaved_bf16(
656
+ a_packed, b_interleaved, sfa, sfb, out,
657
+ float(alpha), int(warps), int(stages),
658
+ )
659
+ return out
660
+
661
+
662
+ def nvfp4_gemm_m256_workspace_size(
663
+ a_packed: torch.Tensor,
664
+ b_packed: torch.Tensor,
665
+ sfa: torch.Tensor,
666
+ sfb: torch.Tensor,
667
+ ) -> int:
668
+ """Return workspace bytes for the SM120 large-M tier.
669
+
670
+ Query and allocate once before CUDA Graph capture.
671
+ """
672
+ return int(ops.nvfp4_gemm_m256_workspace_size(a_packed, b_packed, sfa, sfb))
673
+
674
+
675
+ def nvfp4_gemm_m256_bf16(
676
+ a_packed: torch.Tensor,
677
+ b_packed: torch.Tensor,
678
+ sfa: torch.Tensor,
679
+ sfb: torch.Tensor,
680
+ *,
681
+ workspace: Optional[torch.Tensor] = None,
682
+ alpha: float = 1.0,
683
+ out: Optional[torch.Tensor] = None,
684
+ ) -> torch.Tensor:
685
+ """SM120 M>=512 NVFP4 GEMM with a caller-owned graph-stable workspace."""
686
+ if workspace is None:
687
+ workspace = torch.empty(
688
+ nvfp4_gemm_m256_workspace_size(a_packed, b_packed, sfa, sfb),
689
+ device=a_packed.device,
690
+ dtype=torch.uint8,
691
+ )
692
+ if out is None:
693
+ out = torch.empty(
694
+ (a_packed.shape[0], b_packed.shape[0]),
695
+ device=a_packed.device,
696
+ dtype=torch.bfloat16,
697
+ )
698
+ ops.nvfp4_gemm_m256_bf16(
699
+ a_packed, b_packed, sfa, sfb, workspace, out, float(alpha)
700
+ )
701
+ return out
702
+
703
+
704
  def fp4_w4a16_linear_bf16(
705
  a_packed: torch.Tensor,
706
  b_packed: torch.Tensor,
 
867
  "dequantize_fp4_sfa_fp16",
868
  "e0m3_weight_gemm_fp16",
869
  "fp4_w4a16_linear_bf16",
870
+ "fp4_repack_b_interleaved_sm120",
871
  "fp4_w4a4_gemv_warpsplit_bf16",
872
+ "fp4_w4a4_gemv_warpsplit_interleaved_bf16",
873
  "nvfp4_gemm_bf16",
874
  "nvfp4_gemm_fp16",
875
  "nvfp4_gemm_variant_bf16",
876
  "nvfp4_gemm_nvfp4",
877
+ "nvfp4_gemm_m256_bf16",
878
+ "nvfp4_gemm_m256_workspace_size",
879
  "nvfp4_gemm_geglu_nvfp4_fp16",
880
  "nvfp4_gemm_bias_gelu_nvfp4_fp16",
881
  "nvfp4_gemm_bias_residual_fp16",
build/torch211-cxx11-cu128-x86_64-linux/{_fp4_gemm_cuda_3cf453b.abi3.so → _fp4_gemm_cuda_0a10e1f.abi3.so} RENAMED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:78c2183adc8195e0645609429ddbae0cf3130513c8841b99d0659ca20e71dd8d
3
- size 2093080
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7d86f743a39f009ff82abe0b2cef01f15eafc24ca2c024c450d651ba0e0447e3
3
+ size 2461088
build/torch211-cxx11-cu128-x86_64-linux/_ops.py CHANGED
@@ -1,9 +1,9 @@
1
  import torch
2
- from . import _fp4_gemm_cuda_3cf453b
3
- ops = torch.ops._fp4_gemm_cuda_3cf453b
4
 
5
  def add_op_namespace_prefix(op_name: str):
6
  """
7
  Prefix op by namespace.
8
  """
9
- return f"_fp4_gemm_cuda_3cf453b::{op_name}"
 
1
  import torch
2
+ from . import _fp4_gemm_cuda_0a10e1f
3
+ ops = torch.ops._fp4_gemm_cuda_0a10e1f
4
 
5
  def add_op_namespace_prefix(op_name: str):
6
  """
7
  Prefix op by namespace.
8
  """
9
+ return f"_fp4_gemm_cuda_0a10e1f::{op_name}"
build/torch211-cxx11-cu128-x86_64-linux/metadata.json CHANGED
@@ -1,6 +1,6 @@
1
  {
2
  "name": "fp4-gemm",
3
- "id": "_fp4_gemm_cuda_3cf453b",
4
  "version": 1,
5
  "license": "Apache-2.0",
6
  "python-depends": [],
@@ -13,9 +13,9 @@
13
  "digest": {
14
  "algorithm": "sha256",
15
  "files": {
16
- "__init__.py": "lJLCqFZi1vLuBzLeaNpAAfy/5/sHKv/E70REf7nu6ds=",
17
- "_fp4_gemm_cuda_3cf453b.abi3.so": "eMIYOtyBleBkVglCnduuDPMTBRPIhBuZ0GWcog5x3Y0=",
18
- "_ops.py": "htjsKuSbtwjw+R6PK+Wg4yiVz73RQvTzuJae/Up8mFU=",
19
  "fp4_gemm/__init__.py": "DFYPlrhXwYjEqCl/8n0SmWGZV8NFml5DPhMjKfv98GY="
20
  }
21
  },
@@ -26,7 +26,7 @@
26
  "dirty": false
27
  },
28
  "kernel": {
29
- "sha": "3cf453b28a019b4703dc3a8092eb6badc8f2b18c",
30
  "dirty": false
31
  }
32
  }
 
1
  {
2
  "name": "fp4-gemm",
3
+ "id": "_fp4_gemm_cuda_0a10e1f",
4
  "version": 1,
5
  "license": "Apache-2.0",
6
  "python-depends": [],
 
13
  "digest": {
14
  "algorithm": "sha256",
15
  "files": {
16
+ "__init__.py": "nTluIQWuLs7LotHRt43tTkaA+FufNMoFyUhMfJfbB+c=",
17
+ "_fp4_gemm_cuda_0a10e1f.abi3.so": "fYb3Q6OfAJ/4Kr4LLO8B8V6vwkyiwCTEUNZRug4ER+M=",
18
+ "_ops.py": "dj/4ddT43Gl5g5YOtm2kvHHREJY3sqJsQFxHG3JEBdc=",
19
  "fp4_gemm/__init__.py": "DFYPlrhXwYjEqCl/8n0SmWGZV8NFml5DPhMjKfv98GY="
20
  }
21
  },
 
26
  "dirty": false
27
  },
28
  "kernel": {
29
+ "sha": "0a10e1f3487fa4e673aa18173383d515de28ddd8",
30
  "dirty": false
31
  }
32
  }