liangsu9988 commited on
Commit
bdfe80b
·
1 Parent(s): 2f841ca

Clarify MiniMax MSA Blackwell callable APIs

Browse files
Files changed (2) hide show
  1. CARD.md +113 -153
  2. README.md +113 -153
CARD.md CHANGED
@@ -12,16 +12,14 @@ tags:
12
 
13
  # MiniMaxAI MSA Blackwell
14
 
15
- FlashRT-packaged Blackwell-family extension for MiniMaxAI MSA decode sparse
16
- attention. The upstream MiniMaxAI package is
17
  [`MiniMaxAI/msa`](https://huggingface.co/kernels/MiniMaxAI/msa), which targets
18
- SM100. This package keeps the MiniMax MSA Tensor API style and extends the
19
- decode-sparse path to CUDA compute capability 12.x Blackwell targets.
 
20
 
21
- The package is validated in the FlashRT MiniMax-Spark runtime on DGX Spark /
22
- GB10 and exposes standalone Tensor APIs for use from Python.
23
-
24
- ## Install and Load
25
 
26
  ```python
27
  from kernels import get_kernel
@@ -33,156 +31,113 @@ msa = get_kernel(
33
  )
34
  ```
35
 
36
- ## Current v1 Functions
37
-
38
- This v1 Blackwell package intentionally exposes the subset that has already
39
- been ported and validated on FlashRT's MiniMax-Spark decode path. It is not yet
40
- a drop-in mirror of every public function exported by the upstream
41
- [`MiniMaxAI/msa`](https://huggingface.co/kernels/MiniMaxAI/msa) SM100 package.
42
-
43
- ### `has_native_ops() -> bool`
44
-
45
- Returns whether the native CUDA helper extension was loaded.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
46
 
47
  ```python
48
- assert msa.has_native_ops()
49
- ```
50
 
51
- ### `native_topk_from_scores(score, seq_lens, block_size, topk) -> topk_idx`
52
 
53
- Native CUDA helper that converts precomputed MiniMax MSA block scores to sparse
54
- block ids.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
55
 
56
- - `score`: CUDA `float32`, shape `[heads, batch, max_blocks]`
57
- - `seq_lens`: CUDA `int32`, shape `[batch]`
58
- - `block_size`: currently validated with `128`
59
- - `topk`: currently validated with `16`
60
- - return: CUDA `int32`, shape `[heads, batch, topk]`
61
 
62
  ```python
63
- import torch
64
-
65
- score = torch.randn(64, 1, 256, device="cuda", dtype=torch.float32)
66
- seq_lens = torch.tensor([32768], device="cuda", dtype=torch.int32)
67
- topk_idx = msa.native_topk_from_scores(score, seq_lens, block_size=128, topk=16)
 
 
 
 
 
 
 
68
  ```
69
 
70
- ### `flash_decode_with_topk_idx(...) -> (index_value, topk_idx, real_seq_lens)`
71
-
72
- MiniMax decode indexer. In the common sparse-indexing mode, pass
73
- `disable_index_value=True` and the function returns `(None, topk_idx,
74
- real_seq_lens)`.
75
-
76
- Typical indexer inputs:
77
-
78
- - `q`: CUDA `bfloat16`, shape `[batch, 1, 128]`
79
- - `k_cache`: CUDA `bfloat16`, shape `[ctx, 1, 128]`
80
- - `req_to_token`: CUDA `int32`, shape `[batch, ctx]`
81
- - `seq_lens`: CUDA `int32`, shape `[batch]`
82
- - `slot_ids`: CUDA integer, shape `[batch]`
83
- - output `topk_idx`: CUDA `int32`, shape `[1, batch, topk]`
84
-
85
- ### `flash_decode_with_gqa_share_sparse(...) -> out`
86
-
87
- Block-sparse GQA decode attention. This consumes `topk_idx` from the indexer
88
- and applies the selected sparse blocks to the full GQA attention path.
89
-
90
- Typical MiniMax M3 decode inputs:
91
 
92
- - `q`: CUDA `bfloat16`, shape `[batch, 64, 128]`
93
- - `k_cache`: CUDA `bfloat16`, shape `[ctx, 4, 128]`
94
- - `v_cache`: CUDA `bfloat16`, shape `[ctx, 4, 128]`
95
- - `topk_idx`: CUDA `int32`, shape `[4, batch, 16]`
96
- - return: CUDA `bfloat16`, shape `[batch, 64, 128]`
97
-
98
- ### Reference Functions
99
-
100
- These are provided for local correctness checks:
101
-
102
- - `naive_flash_decode_with_topk_idx(...)`
103
- - `naive_flash_decode_with_gqa_share_sparse(...)`
104
-
105
- ## Upstream API Compatibility Plan
106
-
107
- The upstream `MiniMaxAI/msa` package currently exposes a broader SM100 API
108
- surface:
109
-
110
- | Official `MiniMaxAI/msa` name | v1 status | Planned target |
111
- |---|---|---|
112
- | `sparse_atten_func` | planned | v2+ |
113
- | `sparse_atten_nvfp4_kv_func` | planned | v2+ |
114
- | `sparse_decode_atten_func` | planned | v2 |
115
- | `SparseDecodePagedAttentionWrapper` | planned | v2 |
116
- | `fp4_indexer_block_scores` | planned | v2+ |
117
- | `build_k2q_csr` | planned | v2+ |
118
- | `SparseK2qCsrBuilderSm100` | SM100-specific name; not exported in v1 | v2+ alias decision |
119
- | `Nvfp4QuantizedTensor` | planned | v2+ |
120
- | `quantize_bf16_to_nvfp4_128x4` | planned | v2+ |
121
- | `quantize_kv_bf16_to_nvfp4_128x4` | planned | v2+ |
122
- | `dequantize_nvfp4_128x4_to_bf16` | planned | v2+ |
123
- | `swizzle_nvfp4_scale_to_128x4` | planned | v2+ |
124
- | `nvfp4_global_scale_from_amax` | planned | v2+ |
125
-
126
- Those names are intentionally not exported in v1. The v2 goal is to add a
127
- compatibility layer for the official MiniMaxAI API names where the Blackwell
128
- implementation is ready, starting with the decode path
129
- (`sparse_decode_atten_func` and `SparseDecodePagedAttentionWrapper`), then
130
- expanding to indexing/CSR and NVFP4 helpers after separate correctness
131
- validation.
132
-
133
- Source tracking note: the package source tree includes an `api_status.py`
134
- matrix and tests that keep this v1/v2 boundary explicit. This is a
135
- documentation/introspection aid; it does not make planned official API names
136
- callable before their Blackwell implementations and correctness tests are
137
- added.
138
-
139
- ## Minimal Decode Example
140
 
141
  ```python
142
- import torch
143
- from kernels import get_kernel
144
-
145
- msa = get_kernel("flashrt/MiniMaxAI-msa-blackwell", version=1, trust_remote_code=True)
146
-
147
- batch, hq, hkv, d = 1, 64, 4, 128
148
- ctx, block, topk = 4096, 128, 16
149
- device, dtype = "cuda", torch.bfloat16
150
-
151
- q = torch.randn(batch, hq, d, device=device, dtype=dtype)
152
- k_cache = torch.randn(ctx, hkv, d, device=device, dtype=dtype)
153
- v_cache = torch.randn(ctx, hkv, d, device=device, dtype=dtype)
154
- q_index = torch.randn(batch, 1, d, device=device, dtype=dtype)
155
- k_index = torch.randn(ctx, 1, d, device=device, dtype=dtype)
156
- req_to_token = torch.arange(ctx, device=device, dtype=torch.int32).view(1, -1)
157
- seq_lens = torch.tensor([ctx], device=device, dtype=torch.int32)
158
- slot_ids = torch.zeros(batch, device=device, dtype=torch.int64)
159
-
160
- _, topk_idx, _ = msa.flash_decode_with_topk_idx(
161
- q_index,
162
- None,
163
- k_index,
164
- None,
165
- req_to_token,
166
- seq_lens,
167
- max_seqlen=ctx,
168
- slot_ids=slot_ids,
169
- block_size=block,
170
- topk=topk,
171
- init_blocks=0,
172
- local_blocks=1,
173
- disable_index_value=True,
174
- )
175
 
176
  out = msa.flash_decode_with_gqa_share_sparse(
177
- q,
178
- None,
179
- k_cache,
180
- v_cache,
181
- req_to_token,
182
- seq_lens,
183
- slot_ids,
184
- block,
185
- topk_idx.expand(hkv, batch, topk).contiguous(),
186
  )
187
  ```
188
 
@@ -190,20 +145,25 @@ out = msa.flash_decode_with_gqa_share_sparse(
190
 
191
  - Target family: NVIDIA Blackwell CUDA compute capability 12.x.
192
  - Builder target: CUDA 12.8+ with `cuda-capabilities = ["12.0", "12.1"]`.
193
- - Published artifact metadata: CUDA 12.8 variants currently list `12.0`;
194
- CUDA 13.0/13.2 variants list both `12.0` and `12.1`.
195
  - Validated hardware: DGX Spark / GB10 / SM121.
196
  - Validated MiniMax shape: query heads `64`, KV heads `4`, head dim `128`,
197
- sparse block `128`, top-k blocks `16`.
198
  - Validated context lengths: `128`, `2048`, `4096`, `32768`.
199
  - Correctness gate: cosine similarity `>= 0.999` against paged FP32 PyTorch
200
- references; native score-to-top-k selection matches PyTorch blockmax top-k
201
- set semantics.
 
 
 
 
 
 
 
 
202
 
203
- Implementation note: this package includes a native CUDA top-k helper and a
204
- Blackwell-validated Triton CUDA decode sparse attention fallback. A full native
205
- CUTE/CUDA attention body aligned with upstream `MiniMaxAI/msa` is the next
206
- implementation step.
207
 
208
  Source provenance and validation details are documented in `SYNC.md` and
209
  `VALIDATION.md`.
 
12
 
13
  # MiniMaxAI MSA Blackwell
14
 
15
+ Blackwell-family package for MiniMax MSA decode sparse attention, maintained by
16
+ FlashRT. The upstream package is
17
  [`MiniMaxAI/msa`](https://huggingface.co/kernels/MiniMaxAI/msa), which targets
18
+ SM100. This package extends the decode-sparse path to NVIDIA Blackwell
19
+ compute capability 12.x and has been validated in the FlashRT MiniMax-Spark
20
+ runtime on DGX Spark / GB10 / SM121.
21
 
22
+ ## Load
 
 
 
23
 
24
  ```python
25
  from kernels import get_kernel
 
31
  )
32
  ```
33
 
34
+ ## What You Can Call
35
+
36
+ ### Official MiniMaxAI/msa names
37
+
38
+ | Function/class | Status in this Blackwell package |
39
+ |---|---|
40
+ | `sparse_decode_atten_func` | Available. Blackwell paged BF16/FP16 single-token decode wrapper. |
41
+ | `SparseDecodePagedAttentionWrapper` | Available. `plan(...).run(...)` wrapper for the same decode path. |
42
+ | `build_k2q_csr` | Available. Torch CSR construction fallback. |
43
+ | `SparseK2qCsrBuilderSm100` | Available compatibility class; `build()` delegates to `build_k2q_csr`. |
44
+ | `Nvfp4QuantizedTensor` | Available metadata dataclass. |
45
+ | `quantize_bf16_to_nvfp4_128x4` | Available when Transformer Engine NVFP4 support is installed. |
46
+ | `quantize_kv_bf16_to_nvfp4_128x4` | Available when Transformer Engine NVFP4 support is installed. |
47
+ | `dequantize_nvfp4_128x4_to_bf16` | Available reference dequantizer. |
48
+ | `swizzle_nvfp4_scale_to_128x4` | Available scale-layout helper. |
49
+ | `nvfp4_global_scale_from_amax` | Available scale helper. |
50
+ | `sparse_atten_func` | Exported, but raises `NotImplementedError`: SM100 CSR prefill kernel is not ported here yet. |
51
+ | `sparse_atten_nvfp4_kv_func` | Exported, but raises `NotImplementedError`: SM100 NVFP4 CSR prefill kernel is not ported here yet. |
52
+ | `fp4_indexer_block_scores` | Exported, but raises `NotImplementedError`: SM100 FP4 CUTE indexer is not ported here yet. |
53
+
54
+ ### FlashRT Blackwell helper names
55
+
56
+ These are the direct low-level APIs used by the FlashRT MiniMax-Spark decode
57
+ path:
58
+
59
+ - `flash_decode_with_topk_idx`
60
+ - `flash_decode_with_gqa_share_sparse`
61
+ - `native_topk_from_scores`
62
+ - `has_native_ops`
63
+ - `naive_flash_decode_with_topk_idx`
64
+ - `naive_flash_decode_with_gqa_share_sparse`
65
+ - `get_cu_seqblocks`
66
+ - `robust_allocator`
67
+
68
+ ## Decode Example
69
+
70
+ This example uses the official MiniMax decode-facing name
71
+ `sparse_decode_atten_func`.
72
 
73
  ```python
74
+ import torch
75
+ from kernels import get_kernel
76
 
77
+ msa = get_kernel("flashrt/MiniMaxAI-msa-blackwell", version=1, trust_remote_code=True)
78
 
79
+ B, Hq, Hkv, D = 1, 64, 4, 128
80
+ page_size = 128
81
+ num_pages = 32
82
+ topk = 16
83
+ device = "cuda"
84
+ dtype = torch.bfloat16
85
+
86
+ q = torch.randn(B, Hq, D, device=device, dtype=dtype)
87
+ k = torch.randn(num_pages, Hkv, page_size, D, device=device, dtype=dtype)
88
+ v = torch.randn_like(k)
89
+ page_table = torch.arange(num_pages, device=device, dtype=torch.int32).view(B, -1)
90
+ seqused_k = torch.tensor([num_pages * page_size], device=device, dtype=torch.int32)
91
+ q2k_indices = torch.arange(topk, device=device, dtype=torch.int32).view(1, 1, topk)
92
+ q2k_indices = q2k_indices.expand(Hkv, B, topk).contiguous()
93
+
94
+ out = msa.sparse_decode_atten_func(
95
+ q,
96
+ k,
97
+ v,
98
+ q2k_indices,
99
+ page_table=page_table,
100
+ seqused_k=seqused_k,
101
+ seqlen_q=1,
102
+ max_seqlen_k=num_pages * page_size,
103
+ blk_kv=page_size,
104
+ )
105
+ ```
106
 
107
+ ## Wrapper Example
 
 
 
 
108
 
109
  ```python
110
+ wrapper = msa.SparseDecodePagedAttentionWrapper(blk_kv=128)
111
+ wrapper.plan(
112
+ page_table=page_table,
113
+ seqused_k=seqused_k,
114
+ seqlen_q=1,
115
+ max_seqlen_k=num_pages * page_size,
116
+ q2k_indices=q2k_indices,
117
+ num_qo_heads=Hq,
118
+ num_kv_heads=Hkv,
119
+ head_dim=D,
120
+ )
121
+ out = wrapper.run(q, k, v)
122
  ```
123
 
124
+ ## Direct FlashRT Decode Path
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
125
 
126
+ Use this lower-level path if you already have `topk_idx` in FlashRT's paged KV
127
+ layout.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
128
 
129
  ```python
130
+ q = torch.randn(B, Hq, D, device=device, dtype=dtype)
131
+ k_cache = torch.randn(num_pages * page_size, Hkv, D, device=device, dtype=dtype)
132
+ v_cache = torch.randn_like(k_cache)
133
+ req_to_token = torch.arange(num_pages * page_size, device=device, dtype=torch.int32).view(B, -1)
134
+ seq_lens = torch.tensor([num_pages * page_size], device=device, dtype=torch.int32)
135
+ slot_ids = torch.zeros(B, device=device, dtype=torch.int64)
136
+ topk_idx = torch.arange(topk, device=device, dtype=torch.int32).view(1, 1, topk)
137
+ topk_idx = topk_idx.expand(Hkv, B, topk).contiguous()
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
138
 
139
  out = msa.flash_decode_with_gqa_share_sparse(
140
+ q, None, k_cache, v_cache, req_to_token, seq_lens, slot_ids, page_size, topk_idx
 
 
 
 
 
 
 
 
141
  )
142
  ```
143
 
 
145
 
146
  - Target family: NVIDIA Blackwell CUDA compute capability 12.x.
147
  - Builder target: CUDA 12.8+ with `cuda-capabilities = ["12.0", "12.1"]`.
 
 
148
  - Validated hardware: DGX Spark / GB10 / SM121.
149
  - Validated MiniMax shape: query heads `64`, KV heads `4`, head dim `128`,
150
+ sparse block/page size `128`, top-k blocks `16`.
151
  - Validated context lengths: `128`, `2048`, `4096`, `32768`.
152
  - Correctness gate: cosine similarity `>= 0.999` against paged FP32 PyTorch
153
+ references; official decode wrapper output matches the direct Blackwell
154
+ decode kernel.
155
+
156
+ ## Implementation Notes
157
+
158
+ This package contains:
159
+
160
+ - native CUDA score-to-top-k helper;
161
+ - Blackwell-validated Triton CUDA sparse decode attention;
162
+ - MiniMaxAI/msa-compatible Python API layer for decode, CSR, and NVFP4 helpers.
163
 
164
+ The upstream SM100 CSR prefill attention and FP4 CUTE indexer are not claimed as
165
+ ported in this package yet. Their root names are exported so callers see a clear
166
+ `NotImplementedError` instead of silently getting wrong behavior.
 
167
 
168
  Source provenance and validation details are documented in `SYNC.md` and
169
  `VALIDATION.md`.
README.md CHANGED
@@ -12,16 +12,14 @@ tags:
12
 
13
  # MiniMaxAI MSA Blackwell
14
 
15
- FlashRT-packaged Blackwell-family extension for MiniMaxAI MSA decode sparse
16
- attention. The upstream MiniMaxAI package is
17
  [`MiniMaxAI/msa`](https://huggingface.co/kernels/MiniMaxAI/msa), which targets
18
- SM100. This package keeps the MiniMax MSA Tensor API style and extends the
19
- decode-sparse path to CUDA compute capability 12.x Blackwell targets.
 
20
 
21
- The package is validated in the FlashRT MiniMax-Spark runtime on DGX Spark /
22
- GB10 and exposes standalone Tensor APIs for use from Python.
23
-
24
- ## Install and Load
25
 
26
  ```python
27
  from kernels import get_kernel
@@ -33,156 +31,113 @@ msa = get_kernel(
33
  )
34
  ```
35
 
36
- ## Current v1 Functions
37
-
38
- This v1 Blackwell package intentionally exposes the subset that has already
39
- been ported and validated on FlashRT's MiniMax-Spark decode path. It is not yet
40
- a drop-in mirror of every public function exported by the upstream
41
- [`MiniMaxAI/msa`](https://huggingface.co/kernels/MiniMaxAI/msa) SM100 package.
42
-
43
- ### `has_native_ops() -> bool`
44
-
45
- Returns whether the native CUDA helper extension was loaded.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
46
 
47
  ```python
48
- assert msa.has_native_ops()
49
- ```
50
 
51
- ### `native_topk_from_scores(score, seq_lens, block_size, topk) -> topk_idx`
52
 
53
- Native CUDA helper that converts precomputed MiniMax MSA block scores to sparse
54
- block ids.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
55
 
56
- - `score`: CUDA `float32`, shape `[heads, batch, max_blocks]`
57
- - `seq_lens`: CUDA `int32`, shape `[batch]`
58
- - `block_size`: currently validated with `128`
59
- - `topk`: currently validated with `16`
60
- - return: CUDA `int32`, shape `[heads, batch, topk]`
61
 
62
  ```python
63
- import torch
64
-
65
- score = torch.randn(64, 1, 256, device="cuda", dtype=torch.float32)
66
- seq_lens = torch.tensor([32768], device="cuda", dtype=torch.int32)
67
- topk_idx = msa.native_topk_from_scores(score, seq_lens, block_size=128, topk=16)
 
 
 
 
 
 
 
68
  ```
69
 
70
- ### `flash_decode_with_topk_idx(...) -> (index_value, topk_idx, real_seq_lens)`
71
-
72
- MiniMax decode indexer. In the common sparse-indexing mode, pass
73
- `disable_index_value=True` and the function returns `(None, topk_idx,
74
- real_seq_lens)`.
75
-
76
- Typical indexer inputs:
77
-
78
- - `q`: CUDA `bfloat16`, shape `[batch, 1, 128]`
79
- - `k_cache`: CUDA `bfloat16`, shape `[ctx, 1, 128]`
80
- - `req_to_token`: CUDA `int32`, shape `[batch, ctx]`
81
- - `seq_lens`: CUDA `int32`, shape `[batch]`
82
- - `slot_ids`: CUDA integer, shape `[batch]`
83
- - output `topk_idx`: CUDA `int32`, shape `[1, batch, topk]`
84
-
85
- ### `flash_decode_with_gqa_share_sparse(...) -> out`
86
-
87
- Block-sparse GQA decode attention. This consumes `topk_idx` from the indexer
88
- and applies the selected sparse blocks to the full GQA attention path.
89
-
90
- Typical MiniMax M3 decode inputs:
91
 
92
- - `q`: CUDA `bfloat16`, shape `[batch, 64, 128]`
93
- - `k_cache`: CUDA `bfloat16`, shape `[ctx, 4, 128]`
94
- - `v_cache`: CUDA `bfloat16`, shape `[ctx, 4, 128]`
95
- - `topk_idx`: CUDA `int32`, shape `[4, batch, 16]`
96
- - return: CUDA `bfloat16`, shape `[batch, 64, 128]`
97
-
98
- ### Reference Functions
99
-
100
- These are provided for local correctness checks:
101
-
102
- - `naive_flash_decode_with_topk_idx(...)`
103
- - `naive_flash_decode_with_gqa_share_sparse(...)`
104
-
105
- ## Upstream API Compatibility Plan
106
-
107
- The upstream `MiniMaxAI/msa` package currently exposes a broader SM100 API
108
- surface:
109
-
110
- | Official `MiniMaxAI/msa` name | v1 status | Planned target |
111
- |---|---|---|
112
- | `sparse_atten_func` | planned | v2+ |
113
- | `sparse_atten_nvfp4_kv_func` | planned | v2+ |
114
- | `sparse_decode_atten_func` | planned | v2 |
115
- | `SparseDecodePagedAttentionWrapper` | planned | v2 |
116
- | `fp4_indexer_block_scores` | planned | v2+ |
117
- | `build_k2q_csr` | planned | v2+ |
118
- | `SparseK2qCsrBuilderSm100` | SM100-specific name; not exported in v1 | v2+ alias decision |
119
- | `Nvfp4QuantizedTensor` | planned | v2+ |
120
- | `quantize_bf16_to_nvfp4_128x4` | planned | v2+ |
121
- | `quantize_kv_bf16_to_nvfp4_128x4` | planned | v2+ |
122
- | `dequantize_nvfp4_128x4_to_bf16` | planned | v2+ |
123
- | `swizzle_nvfp4_scale_to_128x4` | planned | v2+ |
124
- | `nvfp4_global_scale_from_amax` | planned | v2+ |
125
-
126
- Those names are intentionally not exported in v1. The v2 goal is to add a
127
- compatibility layer for the official MiniMaxAI API names where the Blackwell
128
- implementation is ready, starting with the decode path
129
- (`sparse_decode_atten_func` and `SparseDecodePagedAttentionWrapper`), then
130
- expanding to indexing/CSR and NVFP4 helpers after separate correctness
131
- validation.
132
-
133
- Source tracking note: the package source tree includes an `api_status.py`
134
- matrix and tests that keep this v1/v2 boundary explicit. This is a
135
- documentation/introspection aid; it does not make planned official API names
136
- callable before their Blackwell implementations and correctness tests are
137
- added.
138
-
139
- ## Minimal Decode Example
140
 
141
  ```python
142
- import torch
143
- from kernels import get_kernel
144
-
145
- msa = get_kernel("flashrt/MiniMaxAI-msa-blackwell", version=1, trust_remote_code=True)
146
-
147
- batch, hq, hkv, d = 1, 64, 4, 128
148
- ctx, block, topk = 4096, 128, 16
149
- device, dtype = "cuda", torch.bfloat16
150
-
151
- q = torch.randn(batch, hq, d, device=device, dtype=dtype)
152
- k_cache = torch.randn(ctx, hkv, d, device=device, dtype=dtype)
153
- v_cache = torch.randn(ctx, hkv, d, device=device, dtype=dtype)
154
- q_index = torch.randn(batch, 1, d, device=device, dtype=dtype)
155
- k_index = torch.randn(ctx, 1, d, device=device, dtype=dtype)
156
- req_to_token = torch.arange(ctx, device=device, dtype=torch.int32).view(1, -1)
157
- seq_lens = torch.tensor([ctx], device=device, dtype=torch.int32)
158
- slot_ids = torch.zeros(batch, device=device, dtype=torch.int64)
159
-
160
- _, topk_idx, _ = msa.flash_decode_with_topk_idx(
161
- q_index,
162
- None,
163
- k_index,
164
- None,
165
- req_to_token,
166
- seq_lens,
167
- max_seqlen=ctx,
168
- slot_ids=slot_ids,
169
- block_size=block,
170
- topk=topk,
171
- init_blocks=0,
172
- local_blocks=1,
173
- disable_index_value=True,
174
- )
175
 
176
  out = msa.flash_decode_with_gqa_share_sparse(
177
- q,
178
- None,
179
- k_cache,
180
- v_cache,
181
- req_to_token,
182
- seq_lens,
183
- slot_ids,
184
- block,
185
- topk_idx.expand(hkv, batch, topk).contiguous(),
186
  )
187
  ```
188
 
@@ -190,20 +145,25 @@ out = msa.flash_decode_with_gqa_share_sparse(
190
 
191
  - Target family: NVIDIA Blackwell CUDA compute capability 12.x.
192
  - Builder target: CUDA 12.8+ with `cuda-capabilities = ["12.0", "12.1"]`.
193
- - Published artifact metadata: CUDA 12.8 variants currently list `12.0`;
194
- CUDA 13.0/13.2 variants list both `12.0` and `12.1`.
195
  - Validated hardware: DGX Spark / GB10 / SM121.
196
  - Validated MiniMax shape: query heads `64`, KV heads `4`, head dim `128`,
197
- sparse block `128`, top-k blocks `16`.
198
  - Validated context lengths: `128`, `2048`, `4096`, `32768`.
199
  - Correctness gate: cosine similarity `>= 0.999` against paged FP32 PyTorch
200
- references; native score-to-top-k selection matches PyTorch blockmax top-k
201
- set semantics.
 
 
 
 
 
 
 
 
202
 
203
- Implementation note: this package includes a native CUDA top-k helper and a
204
- Blackwell-validated Triton CUDA decode sparse attention fallback. A full native
205
- CUTE/CUDA attention body aligned with upstream `MiniMaxAI/msa` is the next
206
- implementation step.
207
 
208
  Source provenance and validation details are documented in `SYNC.md` and
209
  `VALIDATION.md`.
 
12
 
13
  # MiniMaxAI MSA Blackwell
14
 
15
+ Blackwell-family package for MiniMax MSA decode sparse attention, maintained by
16
+ FlashRT. The upstream package is
17
  [`MiniMaxAI/msa`](https://huggingface.co/kernels/MiniMaxAI/msa), which targets
18
+ SM100. This package extends the decode-sparse path to NVIDIA Blackwell
19
+ compute capability 12.x and has been validated in the FlashRT MiniMax-Spark
20
+ runtime on DGX Spark / GB10 / SM121.
21
 
22
+ ## Load
 
 
 
23
 
24
  ```python
25
  from kernels import get_kernel
 
31
  )
32
  ```
33
 
34
+ ## What You Can Call
35
+
36
+ ### Official MiniMaxAI/msa names
37
+
38
+ | Function/class | Status in this Blackwell package |
39
+ |---|---|
40
+ | `sparse_decode_atten_func` | Available. Blackwell paged BF16/FP16 single-token decode wrapper. |
41
+ | `SparseDecodePagedAttentionWrapper` | Available. `plan(...).run(...)` wrapper for the same decode path. |
42
+ | `build_k2q_csr` | Available. Torch CSR construction fallback. |
43
+ | `SparseK2qCsrBuilderSm100` | Available compatibility class; `build()` delegates to `build_k2q_csr`. |
44
+ | `Nvfp4QuantizedTensor` | Available metadata dataclass. |
45
+ | `quantize_bf16_to_nvfp4_128x4` | Available when Transformer Engine NVFP4 support is installed. |
46
+ | `quantize_kv_bf16_to_nvfp4_128x4` | Available when Transformer Engine NVFP4 support is installed. |
47
+ | `dequantize_nvfp4_128x4_to_bf16` | Available reference dequantizer. |
48
+ | `swizzle_nvfp4_scale_to_128x4` | Available scale-layout helper. |
49
+ | `nvfp4_global_scale_from_amax` | Available scale helper. |
50
+ | `sparse_atten_func` | Exported, but raises `NotImplementedError`: SM100 CSR prefill kernel is not ported here yet. |
51
+ | `sparse_atten_nvfp4_kv_func` | Exported, but raises `NotImplementedError`: SM100 NVFP4 CSR prefill kernel is not ported here yet. |
52
+ | `fp4_indexer_block_scores` | Exported, but raises `NotImplementedError`: SM100 FP4 CUTE indexer is not ported here yet. |
53
+
54
+ ### FlashRT Blackwell helper names
55
+
56
+ These are the direct low-level APIs used by the FlashRT MiniMax-Spark decode
57
+ path:
58
+
59
+ - `flash_decode_with_topk_idx`
60
+ - `flash_decode_with_gqa_share_sparse`
61
+ - `native_topk_from_scores`
62
+ - `has_native_ops`
63
+ - `naive_flash_decode_with_topk_idx`
64
+ - `naive_flash_decode_with_gqa_share_sparse`
65
+ - `get_cu_seqblocks`
66
+ - `robust_allocator`
67
+
68
+ ## Decode Example
69
+
70
+ This example uses the official MiniMax decode-facing name
71
+ `sparse_decode_atten_func`.
72
 
73
  ```python
74
+ import torch
75
+ from kernels import get_kernel
76
 
77
+ msa = get_kernel("flashrt/MiniMaxAI-msa-blackwell", version=1, trust_remote_code=True)
78
 
79
+ B, Hq, Hkv, D = 1, 64, 4, 128
80
+ page_size = 128
81
+ num_pages = 32
82
+ topk = 16
83
+ device = "cuda"
84
+ dtype = torch.bfloat16
85
+
86
+ q = torch.randn(B, Hq, D, device=device, dtype=dtype)
87
+ k = torch.randn(num_pages, Hkv, page_size, D, device=device, dtype=dtype)
88
+ v = torch.randn_like(k)
89
+ page_table = torch.arange(num_pages, device=device, dtype=torch.int32).view(B, -1)
90
+ seqused_k = torch.tensor([num_pages * page_size], device=device, dtype=torch.int32)
91
+ q2k_indices = torch.arange(topk, device=device, dtype=torch.int32).view(1, 1, topk)
92
+ q2k_indices = q2k_indices.expand(Hkv, B, topk).contiguous()
93
+
94
+ out = msa.sparse_decode_atten_func(
95
+ q,
96
+ k,
97
+ v,
98
+ q2k_indices,
99
+ page_table=page_table,
100
+ seqused_k=seqused_k,
101
+ seqlen_q=1,
102
+ max_seqlen_k=num_pages * page_size,
103
+ blk_kv=page_size,
104
+ )
105
+ ```
106
 
107
+ ## Wrapper Example
 
 
 
 
108
 
109
  ```python
110
+ wrapper = msa.SparseDecodePagedAttentionWrapper(blk_kv=128)
111
+ wrapper.plan(
112
+ page_table=page_table,
113
+ seqused_k=seqused_k,
114
+ seqlen_q=1,
115
+ max_seqlen_k=num_pages * page_size,
116
+ q2k_indices=q2k_indices,
117
+ num_qo_heads=Hq,
118
+ num_kv_heads=Hkv,
119
+ head_dim=D,
120
+ )
121
+ out = wrapper.run(q, k, v)
122
  ```
123
 
124
+ ## Direct FlashRT Decode Path
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
125
 
126
+ Use this lower-level path if you already have `topk_idx` in FlashRT's paged KV
127
+ layout.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
128
 
129
  ```python
130
+ q = torch.randn(B, Hq, D, device=device, dtype=dtype)
131
+ k_cache = torch.randn(num_pages * page_size, Hkv, D, device=device, dtype=dtype)
132
+ v_cache = torch.randn_like(k_cache)
133
+ req_to_token = torch.arange(num_pages * page_size, device=device, dtype=torch.int32).view(B, -1)
134
+ seq_lens = torch.tensor([num_pages * page_size], device=device, dtype=torch.int32)
135
+ slot_ids = torch.zeros(B, device=device, dtype=torch.int64)
136
+ topk_idx = torch.arange(topk, device=device, dtype=torch.int32).view(1, 1, topk)
137
+ topk_idx = topk_idx.expand(Hkv, B, topk).contiguous()
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
138
 
139
  out = msa.flash_decode_with_gqa_share_sparse(
140
+ q, None, k_cache, v_cache, req_to_token, seq_lens, slot_ids, page_size, topk_idx
 
 
 
 
 
 
 
 
141
  )
142
  ```
143
 
 
145
 
146
  - Target family: NVIDIA Blackwell CUDA compute capability 12.x.
147
  - Builder target: CUDA 12.8+ with `cuda-capabilities = ["12.0", "12.1"]`.
 
 
148
  - Validated hardware: DGX Spark / GB10 / SM121.
149
  - Validated MiniMax shape: query heads `64`, KV heads `4`, head dim `128`,
150
+ sparse block/page size `128`, top-k blocks `16`.
151
  - Validated context lengths: `128`, `2048`, `4096`, `32768`.
152
  - Correctness gate: cosine similarity `>= 0.999` against paged FP32 PyTorch
153
+ references; official decode wrapper output matches the direct Blackwell
154
+ decode kernel.
155
+
156
+ ## Implementation Notes
157
+
158
+ This package contains:
159
+
160
+ - native CUDA score-to-top-k helper;
161
+ - Blackwell-validated Triton CUDA sparse decode attention;
162
+ - MiniMaxAI/msa-compatible Python API layer for decode, CSR, and NVFP4 helpers.
163
 
164
+ The upstream SM100 CSR prefill attention and FP4 CUTE indexer are not claimed as
165
+ ported in this package yet. Their root names are exported so callers see a clear
166
+ `NotImplementedError` instead of silently getting wrong behavior.
 
167
 
168
  Source provenance and validation details are documented in `SYNC.md` and
169
  `VALIDATION.md`.