liangsu9988 commited on
Commit
3cb6c08
·
1 Parent(s): 8bd19f6

Add MiniMax MSA Blackwell kernel card

Browse files
Files changed (2) hide show
  1. CARD.md +170 -0
  2. README.md +170 -0
CARD.md ADDED
@@ -0,0 +1,170 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: kernels
3
+ license: apache-2.0
4
+ tags:
5
+ - cuda
6
+ - triton
7
+ - native-cuda
8
+ - minimax
9
+ - sparse-attention
10
+ - blackwell
11
+ ---
12
+
13
+ # MiniMaxAI MSA Blackwell
14
+
15
+ FlashRT-packaged Blackwell-family extension for MiniMaxAI MSA decode sparse
16
+ attention. The upstream MiniMaxAI package is
17
+ [`MiniMaxAI/msa`](https://huggingface.co/kernels/MiniMaxAI/msa), which targets
18
+ SM100. This package keeps the MiniMax MSA Tensor API style and extends the
19
+ decode-sparse path to CUDA compute capability 12.x Blackwell targets.
20
+
21
+ The package is validated in the FlashRT MiniMax-Spark runtime on DGX Spark /
22
+ GB10 and exposes standalone Tensor APIs for use from Python.
23
+
24
+ ## Install and Load
25
+
26
+ ```python
27
+ from kernels import get_kernel
28
+
29
+ msa = get_kernel(
30
+ "flashrt/MiniMaxAI-msa-blackwell",
31
+ version=1,
32
+ trust_remote_code=True,
33
+ )
34
+ ```
35
+
36
+ ## Functions
37
+
38
+ ### `has_native_ops() -> bool`
39
+
40
+ Returns whether the native CUDA helper extension was loaded.
41
+
42
+ ```python
43
+ assert msa.has_native_ops()
44
+ ```
45
+
46
+ ### `native_topk_from_scores(score, seq_lens, block_size, topk) -> topk_idx`
47
+
48
+ Native CUDA helper that converts precomputed MiniMax MSA block scores to sparse
49
+ block ids.
50
+
51
+ - `score`: CUDA `float32`, shape `[heads, batch, max_blocks]`
52
+ - `seq_lens`: CUDA `int32`, shape `[batch]`
53
+ - `block_size`: currently validated with `128`
54
+ - `topk`: currently validated with `16`
55
+ - return: CUDA `int32`, shape `[heads, batch, topk]`
56
+
57
+ ```python
58
+ import torch
59
+
60
+ score = torch.randn(64, 1, 256, device="cuda", dtype=torch.float32)
61
+ seq_lens = torch.tensor([32768], device="cuda", dtype=torch.int32)
62
+ topk_idx = msa.native_topk_from_scores(score, seq_lens, block_size=128, topk=16)
63
+ ```
64
+
65
+ ### `flash_decode_with_topk_idx(...) -> (index_value, topk_idx, real_seq_lens)`
66
+
67
+ MiniMax decode indexer. In the common sparse-indexing mode, pass
68
+ `disable_index_value=True` and the function returns `(None, topk_idx,
69
+ real_seq_lens)`.
70
+
71
+ Typical indexer inputs:
72
+
73
+ - `q`: CUDA `bfloat16`, shape `[batch, 1, 128]`
74
+ - `k_cache`: CUDA `bfloat16`, shape `[ctx, 1, 128]`
75
+ - `req_to_token`: CUDA `int32`, shape `[batch, ctx]`
76
+ - `seq_lens`: CUDA `int32`, shape `[batch]`
77
+ - `slot_ids`: CUDA integer, shape `[batch]`
78
+ - output `topk_idx`: CUDA `int32`, shape `[1, batch, topk]`
79
+
80
+ ### `flash_decode_with_gqa_share_sparse(...) -> out`
81
+
82
+ Block-sparse GQA decode attention. This consumes `topk_idx` from the indexer
83
+ and applies the selected sparse blocks to the full GQA attention path.
84
+
85
+ Typical MiniMax M3 decode inputs:
86
+
87
+ - `q`: CUDA `bfloat16`, shape `[batch, 64, 128]`
88
+ - `k_cache`: CUDA `bfloat16`, shape `[ctx, 4, 128]`
89
+ - `v_cache`: CUDA `bfloat16`, shape `[ctx, 4, 128]`
90
+ - `topk_idx`: CUDA `int32`, shape `[4, batch, 16]`
91
+ - return: CUDA `bfloat16`, shape `[batch, 64, 128]`
92
+
93
+ ### Reference Functions
94
+
95
+ These are provided for local correctness checks:
96
+
97
+ - `naive_flash_decode_with_topk_idx(...)`
98
+ - `naive_flash_decode_with_gqa_share_sparse(...)`
99
+
100
+ ## Minimal Decode Example
101
+
102
+ ```python
103
+ import torch
104
+ from kernels import get_kernel
105
+
106
+ msa = get_kernel("flashrt/MiniMaxAI-msa-blackwell", version=1, trust_remote_code=True)
107
+
108
+ batch, hq, hkv, d = 1, 64, 4, 128
109
+ ctx, block, topk = 4096, 128, 16
110
+ device, dtype = "cuda", torch.bfloat16
111
+
112
+ q = torch.randn(batch, hq, d, device=device, dtype=dtype)
113
+ k_cache = torch.randn(ctx, hkv, d, device=device, dtype=dtype)
114
+ v_cache = torch.randn(ctx, hkv, d, device=device, dtype=dtype)
115
+ q_index = torch.randn(batch, 1, d, device=device, dtype=dtype)
116
+ k_index = torch.randn(ctx, 1, d, device=device, dtype=dtype)
117
+ req_to_token = torch.arange(ctx, device=device, dtype=torch.int32).view(1, -1)
118
+ seq_lens = torch.tensor([ctx], device=device, dtype=torch.int32)
119
+ slot_ids = torch.zeros(batch, device=device, dtype=torch.int64)
120
+
121
+ _, topk_idx, _ = msa.flash_decode_with_topk_idx(
122
+ q_index,
123
+ None,
124
+ k_index,
125
+ None,
126
+ req_to_token,
127
+ seq_lens,
128
+ max_seqlen=ctx,
129
+ slot_ids=slot_ids,
130
+ block_size=block,
131
+ topk=topk,
132
+ init_blocks=0,
133
+ local_blocks=1,
134
+ disable_index_value=True,
135
+ )
136
+
137
+ out = msa.flash_decode_with_gqa_share_sparse(
138
+ q,
139
+ None,
140
+ k_cache,
141
+ v_cache,
142
+ req_to_token,
143
+ seq_lens,
144
+ slot_ids,
145
+ block,
146
+ topk_idx.expand(hkv, batch, topk).contiguous(),
147
+ )
148
+ ```
149
+
150
+ ## Scope
151
+
152
+ - Target family: NVIDIA Blackwell CUDA compute capability 12.x.
153
+ - Builder target: CUDA 12.8+ with `cuda-capabilities = ["12.0", "12.1"]`.
154
+ - Published artifact metadata: CUDA 12.8 variants currently list `12.0`;
155
+ CUDA 13.0/13.2 variants list both `12.0` and `12.1`.
156
+ - Validated hardware: DGX Spark / GB10 / SM121.
157
+ - Validated MiniMax shape: query heads `64`, KV heads `4`, head dim `128`,
158
+ sparse block `128`, top-k blocks `16`.
159
+ - Validated context lengths: `128`, `2048`, `4096`, `32768`.
160
+ - Correctness gate: cosine similarity `>= 0.999` against paged FP32 PyTorch
161
+ references; native score-to-top-k selection matches PyTorch blockmax top-k
162
+ set semantics.
163
+
164
+ Implementation note: this package includes a native CUDA top-k helper and a
165
+ Blackwell-validated Triton CUDA decode sparse attention fallback. A full native
166
+ CUTE/CUDA attention body aligned with upstream `MiniMaxAI/msa` is the next
167
+ implementation step.
168
+
169
+ Source provenance and validation details are documented in `SYNC.md` and
170
+ `VALIDATION.md`.
README.md ADDED
@@ -0,0 +1,170 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: kernels
3
+ license: apache-2.0
4
+ tags:
5
+ - cuda
6
+ - triton
7
+ - native-cuda
8
+ - minimax
9
+ - sparse-attention
10
+ - blackwell
11
+ ---
12
+
13
+ # MiniMaxAI MSA Blackwell
14
+
15
+ FlashRT-packaged Blackwell-family extension for MiniMaxAI MSA decode sparse
16
+ attention. The upstream MiniMaxAI package is
17
+ [`MiniMaxAI/msa`](https://huggingface.co/kernels/MiniMaxAI/msa), which targets
18
+ SM100. This package keeps the MiniMax MSA Tensor API style and extends the
19
+ decode-sparse path to CUDA compute capability 12.x Blackwell targets.
20
+
21
+ The package is validated in the FlashRT MiniMax-Spark runtime on DGX Spark /
22
+ GB10 and exposes standalone Tensor APIs for use from Python.
23
+
24
+ ## Install and Load
25
+
26
+ ```python
27
+ from kernels import get_kernel
28
+
29
+ msa = get_kernel(
30
+ "flashrt/MiniMaxAI-msa-blackwell",
31
+ version=1,
32
+ trust_remote_code=True,
33
+ )
34
+ ```
35
+
36
+ ## Functions
37
+
38
+ ### `has_native_ops() -> bool`
39
+
40
+ Returns whether the native CUDA helper extension was loaded.
41
+
42
+ ```python
43
+ assert msa.has_native_ops()
44
+ ```
45
+
46
+ ### `native_topk_from_scores(score, seq_lens, block_size, topk) -> topk_idx`
47
+
48
+ Native CUDA helper that converts precomputed MiniMax MSA block scores to sparse
49
+ block ids.
50
+
51
+ - `score`: CUDA `float32`, shape `[heads, batch, max_blocks]`
52
+ - `seq_lens`: CUDA `int32`, shape `[batch]`
53
+ - `block_size`: currently validated with `128`
54
+ - `topk`: currently validated with `16`
55
+ - return: CUDA `int32`, shape `[heads, batch, topk]`
56
+
57
+ ```python
58
+ import torch
59
+
60
+ score = torch.randn(64, 1, 256, device="cuda", dtype=torch.float32)
61
+ seq_lens = torch.tensor([32768], device="cuda", dtype=torch.int32)
62
+ topk_idx = msa.native_topk_from_scores(score, seq_lens, block_size=128, topk=16)
63
+ ```
64
+
65
+ ### `flash_decode_with_topk_idx(...) -> (index_value, topk_idx, real_seq_lens)`
66
+
67
+ MiniMax decode indexer. In the common sparse-indexing mode, pass
68
+ `disable_index_value=True` and the function returns `(None, topk_idx,
69
+ real_seq_lens)`.
70
+
71
+ Typical indexer inputs:
72
+
73
+ - `q`: CUDA `bfloat16`, shape `[batch, 1, 128]`
74
+ - `k_cache`: CUDA `bfloat16`, shape `[ctx, 1, 128]`
75
+ - `req_to_token`: CUDA `int32`, shape `[batch, ctx]`
76
+ - `seq_lens`: CUDA `int32`, shape `[batch]`
77
+ - `slot_ids`: CUDA integer, shape `[batch]`
78
+ - output `topk_idx`: CUDA `int32`, shape `[1, batch, topk]`
79
+
80
+ ### `flash_decode_with_gqa_share_sparse(...) -> out`
81
+
82
+ Block-sparse GQA decode attention. This consumes `topk_idx` from the indexer
83
+ and applies the selected sparse blocks to the full GQA attention path.
84
+
85
+ Typical MiniMax M3 decode inputs:
86
+
87
+ - `q`: CUDA `bfloat16`, shape `[batch, 64, 128]`
88
+ - `k_cache`: CUDA `bfloat16`, shape `[ctx, 4, 128]`
89
+ - `v_cache`: CUDA `bfloat16`, shape `[ctx, 4, 128]`
90
+ - `topk_idx`: CUDA `int32`, shape `[4, batch, 16]`
91
+ - return: CUDA `bfloat16`, shape `[batch, 64, 128]`
92
+
93
+ ### Reference Functions
94
+
95
+ These are provided for local correctness checks:
96
+
97
+ - `naive_flash_decode_with_topk_idx(...)`
98
+ - `naive_flash_decode_with_gqa_share_sparse(...)`
99
+
100
+ ## Minimal Decode Example
101
+
102
+ ```python
103
+ import torch
104
+ from kernels import get_kernel
105
+
106
+ msa = get_kernel("flashrt/MiniMaxAI-msa-blackwell", version=1, trust_remote_code=True)
107
+
108
+ batch, hq, hkv, d = 1, 64, 4, 128
109
+ ctx, block, topk = 4096, 128, 16
110
+ device, dtype = "cuda", torch.bfloat16
111
+
112
+ q = torch.randn(batch, hq, d, device=device, dtype=dtype)
113
+ k_cache = torch.randn(ctx, hkv, d, device=device, dtype=dtype)
114
+ v_cache = torch.randn(ctx, hkv, d, device=device, dtype=dtype)
115
+ q_index = torch.randn(batch, 1, d, device=device, dtype=dtype)
116
+ k_index = torch.randn(ctx, 1, d, device=device, dtype=dtype)
117
+ req_to_token = torch.arange(ctx, device=device, dtype=torch.int32).view(1, -1)
118
+ seq_lens = torch.tensor([ctx], device=device, dtype=torch.int32)
119
+ slot_ids = torch.zeros(batch, device=device, dtype=torch.int64)
120
+
121
+ _, topk_idx, _ = msa.flash_decode_with_topk_idx(
122
+ q_index,
123
+ None,
124
+ k_index,
125
+ None,
126
+ req_to_token,
127
+ seq_lens,
128
+ max_seqlen=ctx,
129
+ slot_ids=slot_ids,
130
+ block_size=block,
131
+ topk=topk,
132
+ init_blocks=0,
133
+ local_blocks=1,
134
+ disable_index_value=True,
135
+ )
136
+
137
+ out = msa.flash_decode_with_gqa_share_sparse(
138
+ q,
139
+ None,
140
+ k_cache,
141
+ v_cache,
142
+ req_to_token,
143
+ seq_lens,
144
+ slot_ids,
145
+ block,
146
+ topk_idx.expand(hkv, batch, topk).contiguous(),
147
+ )
148
+ ```
149
+
150
+ ## Scope
151
+
152
+ - Target family: NVIDIA Blackwell CUDA compute capability 12.x.
153
+ - Builder target: CUDA 12.8+ with `cuda-capabilities = ["12.0", "12.1"]`.
154
+ - Published artifact metadata: CUDA 12.8 variants currently list `12.0`;
155
+ CUDA 13.0/13.2 variants list both `12.0` and `12.1`.
156
+ - Validated hardware: DGX Spark / GB10 / SM121.
157
+ - Validated MiniMax shape: query heads `64`, KV heads `4`, head dim `128`,
158
+ sparse block `128`, top-k blocks `16`.
159
+ - Validated context lengths: `128`, `2048`, `4096`, `32768`.
160
+ - Correctness gate: cosine similarity `>= 0.999` against paged FP32 PyTorch
161
+ references; native score-to-top-k selection matches PyTorch blockmax top-k
162
+ set semantics.
163
+
164
+ Implementation note: this package includes a native CUDA top-k helper and a
165
+ Blackwell-validated Triton CUDA decode sparse attention fallback. A full native
166
+ CUTE/CUDA attention body aligned with upstream `MiniMaxAI/msa` is the next
167
+ implementation step.
168
+
169
+ Source provenance and validation details are documented in `SYNC.md` and
170
+ `VALIDATION.md`.