YMRohit commited on
Commit
0118c5e
·
verified ·
1 Parent(s): d434d4d

Add in-Space offgrid/local mode (ZeroGPU H200 + in-process referee)

Browse files
Files changed (1) hide show
  1. referee/specs.py +1066 -0
referee/specs.py ADDED
@@ -0,0 +1,1066 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """OUROBOROS op suite — the SPECS the proposer writes kernels for.
2
+
3
+ Mirror of `sec_sqli/discovery_specialist`'s environment definition (`OBJ`/`PARAM`/
4
+ `ENVS`), but the "environment" here is a fusion op whose ground truth is a PyTorch
5
+ reference and whose grounded reward is a measured wall-clock speedup.
6
+
7
+ Each `OpSpec` is the immutable problem statement for one op:
8
+ - `reference(*inputs) -> Tensor` : the PyTorch ground truth (correctness oracle).
9
+ - `make_inputs(rng) -> tuple` : ADVERSARIAL randomized inputs — shape, dtype AND
10
+ magnitude vary, so a kernel that is only correct on
11
+ benign N(0,1) inputs (e.g. softmax with no
12
+ max-subtraction) FAILS. This is the GPU analog of
13
+ dvwa_oracle's benign-baseline / anti-pattern-match
14
+ negative controls.
15
+ - `tol(dtype)` : rtol/atol DERIVED from fp accumulation, never
16
+ hand-tuned to make a kernel pass.
17
+ - `signature_hint` : goes into the proposer prompt.
18
+ - The honest baselines (eager + torch.compile) are built by the harness, not stored
19
+ here, so the same `reference` defines both ground truth and the bar.
20
+
21
+ Scope is deliberately HARD-narrowed to FUSION wins (ops where eager launches several
22
+ kernels and Triton can fuse them). We do NOT try to beat cuBLAS at dense GEMM — that is
23
+ the losing-the-window trap called out in the brief.
24
+
25
+ A candidate KERNEL is a Python source string that, when exec'd in a namespace already
26
+ holding {torch, triton, tl}, defines a callable `run(*inputs) -> Tensor` matching the
27
+ op's reference signature. The harness is the only thing that ever runs it.
28
+ """
29
+ from __future__ import annotations
30
+
31
+ from dataclasses import dataclass, field
32
+ from typing import Callable
33
+
34
+ import torch
35
+
36
+ # ----------------------------------------------------------------------------------------
37
+ # Tolerances DERIVED from dtype accumulation behaviour (not tuned to pass).
38
+ # A reduction over D elements accumulates ~sqrt(D) * eps_machine relative error; we give a
39
+ # comfortable but principled envelope per storage dtype. Internals always accumulate in fp32.
40
+ # ----------------------------------------------------------------------------------------
41
+ _TOL = {
42
+ torch.float32: (2e-4, 2e-5),
43
+ torch.float16: (2e-2, 2e-3),
44
+ torch.bfloat16: (3e-2, 8e-3),
45
+ }
46
+
47
+
48
+ def tol_for(dtype: torch.dtype) -> tuple[float, float]:
49
+ return _TOL.get(dtype, (2e-2, 2e-3))
50
+
51
+
52
+ # ----------------------------------------------------------------------------------------
53
+ # Randomized regimes. `make_inputs` must sweep these so correctness is ADVERSARIAL: a
54
+ # kernel has to survive large magnitudes and low-precision dtypes, not just the easy case.
55
+ # ----------------------------------------------------------------------------------------
56
+ _DTYPES = [torch.float16, torch.bfloat16, torch.float32]
57
+ _SCALES = [1.0, 8.0, 64.0] # magnitude sweep — exposes overflow / missing max-sub
58
+ _ROWLEN = [128, 512, 1024, 4096, 4097, 8192] # incl. non-power-of-2 (masking correctness)
59
+ _NROWS = [8, 64, 1000, 4096]
60
+
61
+
62
+ def _pick(rng, xs):
63
+ return xs[rng.randrange(len(xs))]
64
+
65
+
66
+ def _randn(rng, shape, dtype, scale, dev="cuda"):
67
+ g = torch.Generator(device=dev).manual_seed(rng.randrange(2**31))
68
+ return (torch.randn(shape, generator=g, device=dev, dtype=torch.float32) * scale).to(dtype)
69
+
70
+
71
+ # ----------------------------------------------------------------------------------------
72
+ # References (ground truth). Each upcasts to fp32 for the numerically-sensitive reduction
73
+ # exactly as a correct kernel must — so allclose tests the FUSION, not a dtype mismatch.
74
+ # ----------------------------------------------------------------------------------------
75
+ def _rmsnorm_ref(x, w, eps: float = 1e-6):
76
+ xf = x.float()
77
+ rms = torch.rsqrt(xf.pow(2).mean(-1, keepdim=True) + eps)
78
+ return (xf * rms).to(x.dtype) * w
79
+
80
+
81
+ def _softmax_ref(x):
82
+ return torch.softmax(x.float(), dim=-1).to(x.dtype)
83
+
84
+
85
+ def _swiglu_ref(gate, up):
86
+ # the FFN activation fusion: SiLU(gate) * up (elementwise, multi-launch in eager)
87
+ return (torch.nn.functional.silu(gate.float()) * up.float()).to(gate.dtype)
88
+
89
+
90
+ def _add_rmsnorm_ref(x, residual, w, eps: float = 1e-6):
91
+ h = (x.float() + residual.float())
92
+ rms = torch.rsqrt(h.pow(2).mean(-1, keepdim=True) + eps)
93
+ return (h * rms).to(x.dtype) * w
94
+
95
+
96
+ def _rope_ref(x, cos, sin):
97
+ # LLaMA "rotate_half" RoPE: out = x*cos + rotate_half(x)*sin,
98
+ # rotate_half(x) = cat(-x[D/2:], x[:D/2]). Eager: a cat (alloc) + 2 muls + add (multi-launch).
99
+ D = x.shape[-1]; h = D // 2
100
+ xf = x.float()
101
+ rot = torch.cat([-xf[..., h:], xf[..., :h]], dim=-1)
102
+ return (xf * cos.float() + rot * sin.float()).to(x.dtype)
103
+
104
+
105
+ def _layernorm_ref(x, w, b, eps: float = 1e-5):
106
+ # full LayerNorm: subtract mean, divide by std, affine. Two reductions (mean then var) +
107
+ # affine — eager launches several kernels. Distinct from RMSNorm (mean-subtraction + bias).
108
+ xf = x.float()
109
+ mu = xf.mean(-1, keepdim=True)
110
+ xc = xf - mu
111
+ var = (xc * xc).mean(-1, keepdim=True)
112
+ return (xc * torch.rsqrt(var + eps) * w.float() + b.float()).to(x.dtype)
113
+
114
+
115
+ def _add_layernorm_ref(x, residual, w, b, eps: float = 1e-5):
116
+ h = x.float() + residual.float()
117
+ mu = h.mean(-1, keepdim=True)
118
+ hc = h - mu
119
+ var = (hc * hc).mean(-1, keepdim=True)
120
+ return (hc * torch.rsqrt(var + eps) * w.float() + b.float()).to(x.dtype)
121
+
122
+
123
+ _GELU_C = 0.7978845608028654 # sqrt(2/pi)
124
+
125
+
126
+ def _geglu_ref(gate, up):
127
+ # GeGLU: gelu_tanh(gate) * up (the FFN gate with GELU instead of SiLU). tanh approximation.
128
+ g = gate.float()
129
+ gelu = 0.5 * g * (1.0 + torch.tanh(_GELU_C * (g + 0.044715 * g * g * g)))
130
+ return (gelu * up.float()).to(gate.dtype)
131
+
132
+
133
+ def _qknorm_rope_ref(x, w, cos, sin, eps: float = 1e-6):
134
+ # FUSION CHAIN (reduction -> gather): per-head RMSNorm then RoPE — the Qwen-style QK-norm.
135
+ # The composed reference; max-autotune fuses the chain too, so beating it means the model
136
+ # found a schedule inductor's search missed (keeping the rms scale in-register, no
137
+ # intermediate materialized).
138
+ xf = x.float()
139
+ rms = torch.rsqrt(xf.pow(2).mean(-1, keepdim=True) + eps)
140
+ n = xf * rms * w.float()
141
+ D = x.shape[-1]; h = D // 2
142
+ rot = torch.cat([-n[..., h:], n[..., :h]], dim=-1)
143
+ return (n * cos.float() + rot * sin.float()).to(x.dtype)
144
+
145
+
146
+ # ---- comprehensive suite: activations, reductions, softmax family, chains, quant ----------
147
+ def _gelu_ref(x):
148
+ g = x.float()
149
+ return (0.5 * g * (1.0 + torch.tanh(_GELU_C * (g + 0.044715 * g * g * g)))).to(x.dtype)
150
+
151
+
152
+ def _silu_ref(x):
153
+ g = x.float()
154
+ return (g * torch.sigmoid(g)).to(x.dtype)
155
+
156
+
157
+ def _relu2_ref(x):
158
+ g = torch.relu(x.float())
159
+ return (g * g).to(x.dtype)
160
+
161
+
162
+ def _bias_gelu_ref(x, bias):
163
+ g = x.float() + bias.float()
164
+ return (0.5 * g * (1.0 + torch.tanh(_GELU_C * (g + 0.044715 * g * g * g)))).to(x.dtype)
165
+
166
+
167
+ def _reglu_ref(gate, up):
168
+ return (torch.relu(gate.float()) * up.float()).to(gate.dtype)
169
+
170
+
171
+ def _l2norm_ref(x, eps: float = 1e-6):
172
+ xf = x.float()
173
+ return (xf * torch.rsqrt(xf.pow(2).sum(-1, keepdim=True) + eps)).to(x.dtype)
174
+
175
+
176
+ def _log_softmax_ref(x):
177
+ return torch.log_softmax(x.float(), dim=-1).to(x.dtype)
178
+
179
+
180
+ def _softmax_scale_ref(x, scale):
181
+ return torch.softmax(x.float() * scale.float(), dim=-1).to(x.dtype)
182
+
183
+
184
+ def _layernorm_gelu_ref(x, w, b, eps: float = 1e-5):
185
+ xf = x.float()
186
+ mu = xf.mean(-1, keepdim=True)
187
+ xc = xf - mu
188
+ var = (xc * xc).mean(-1, keepdim=True)
189
+ ln = xc * torch.rsqrt(var + eps) * w.float() + b.float()
190
+ return (0.5 * ln * (1.0 + torch.tanh(_GELU_C * (ln + 0.044715 * ln * ln * ln)))).to(x.dtype)
191
+
192
+
193
+ def _add_rmsnorm_rope_ref(x, residual, w, cos, sin, eps: float = 1e-6):
194
+ h = x.float() + residual.float()
195
+ n = h * torch.rsqrt(h.pow(2).mean(-1, keepdim=True) + eps) * w.float()
196
+ D = x.shape[-1]; hd = D // 2
197
+ rot = torch.cat([-n[..., hd:], n[..., :hd]], dim=-1)
198
+ return (n * cos.float() + rot * sin.float()).to(x.dtype)
199
+
200
+
201
+ def _dequant_int8_ref(q, scale):
202
+ # per-row int8 weight dequantization: out = q * scale (memory-bound, NON-GEMM)
203
+ return (q.float() * scale.float()).to(torch.float16)
204
+
205
+
206
+ # ---- V2 standalone additions: real LLM ops beyond the chain grammar ----------------------
207
+ _SOFTCAP = 30.0
208
+
209
+
210
+ def _softcap_softmax_ref(x):
211
+ # Gemma2-style logit softcapping then softmax: softmax(cap * tanh(x / cap)).
212
+ t = _SOFTCAP * torch.tanh(x.float() / _SOFTCAP)
213
+ return torch.softmax(t, dim=-1).to(x.dtype)
214
+
215
+
216
+ def _rmsnorm_gemma_ref(x, w, eps: float = 1e-6):
217
+ # Gemma-style RMSNorm: scale by (1 + w), not w. The +1 is the classic silent-wrongness trap.
218
+ xf = x.float()
219
+ rms = torch.rsqrt(xf.pow(2).mean(-1, keepdim=True) + eps)
220
+ return (xf * rms * (1.0 + w.float())).to(x.dtype)
221
+
222
+
223
+ def _glu_ref(gate, up):
224
+ # the ORIGINAL gated linear unit: sigmoid(gate) * up
225
+ return (torch.sigmoid(gate.float()) * up.float()).to(gate.dtype)
226
+
227
+
228
+ def _rope_interleaved_ref(x, cos, sin):
229
+ # GPT-J / interleaved RoPE: pairs (x[2i], x[2i+1]) rotated by (cos[i], sin[i]).
230
+ xf = x.float()
231
+ x1, x2 = xf[..., 0::2], xf[..., 1::2]
232
+ c, s = cos.float(), sin.float()
233
+ out = torch.empty_like(xf)
234
+ out[..., 0::2] = x1 * c - x2 * s
235
+ out[..., 1::2] = x2 * c + x1 * s
236
+ return out.to(x.dtype)
237
+
238
+
239
+ def _cross_entropy_ref(x, tgt):
240
+ # fused per-row cross-entropy (the Liger flagship fusion): -log_softmax(x)[tgt], no reduction.
241
+ return torch.nn.functional.cross_entropy(x.float(), tgt, reduction="none").to(x.dtype)
242
+
243
+
244
+ # ---- INVENTION suite (V2.7): problem classes the model was never trained on --------------
245
+ def _cumsum_ref(x):
246
+ # row-wise inclusive prefix sum — a SCAN, not a reduction: a different parallel
247
+ # algorithm class (carry propagation across blocks) from everything in the suite.
248
+ return torch.cumsum(x.float(), dim=-1).to(x.dtype)
249
+
250
+
251
+ def _entropy_ref(x):
252
+ # per-row Shannon entropy of softmax(x), fused from logits: H = lse(x) - sum(x*p).
253
+ xf = x.float()
254
+ lse = torch.logsumexp(xf, dim=-1, keepdim=True)
255
+ p = torch.exp(xf - lse)
256
+ return (lse.squeeze(-1) - (xf * p).sum(-1)).to(x.dtype)
257
+
258
+
259
+ def _kl_div_ref(x, y):
260
+ # per-row KL(softmax(x) || softmax(y)) from raw logits — the distillation op.
261
+ # DOUBLE logsumexp fusion; both must be max-subtracted.
262
+ xf, yf = x.float(), y.float()
263
+ lx = xf - torch.logsumexp(xf, dim=-1, keepdim=True)
264
+ ly = yf - torch.logsumexp(yf, dim=-1, keepdim=True)
265
+ return (torch.exp(lx) * (lx - ly)).sum(-1).to(x.dtype)
266
+
267
+
268
+ # ----------------------------------------------------------------------------------------
269
+ @dataclass
270
+ class OpSpec:
271
+ name: str
272
+ reference: Callable
273
+ make_inputs: Callable # rng -> tuple[Tensor, ...] (ADVERSARIAL random, for correctness)
274
+ bench_inputs: Callable # () -> tuple[Tensor, ...] (FIXED large fp16, for timing)
275
+ stress_inputs: Callable # () -> list[tuple] GUARANTEED killer cases (high-scale fp16/bf16,
276
+ # odd N). Run on EVERY eval so a wrong kernel is
277
+ # caught deterministically, never by seed luck.
278
+ signature_hint: str
279
+ notes: str = ""
280
+ extra: dict = field(default_factory=dict)
281
+ # Per-op tolerance override — DERIVED from the op's numerics (documented at the spec),
282
+ # never tuned to make a kernel pass. Needed where the global elementwise envelope is
283
+ # the wrong yardstick (e.g. scans: error tracks the running-path magnitude; entropy:
284
+ # the REFERENCE itself carries catastrophic cancellation).
285
+ tol_override: dict = field(default_factory=dict)
286
+
287
+ def tol(self, dtype: torch.dtype) -> tuple[float, float]:
288
+ return self.tol_override.get(dtype, tol_for(dtype))
289
+
290
+
291
+ # Guaranteed-hard cases: large magnitude (overflow trap), low precision, NON-power-of-2 row
292
+ # length (masking trap), small odd row count. These are run on every correctness check so the
293
+ # negative controls (no max-subtract / no rsqrt) fail DETERMINISTICALLY, not probabilistically.
294
+ def _stress_1tensor(N=4097, M=37):
295
+ return [(_fixed((M, N), torch.float16, 64.0, seed=91),),
296
+ (_fixed((M, N), torch.bfloat16, 64.0, seed=92),),
297
+ (_fixed((3, 8192), torch.float16, 32.0, seed=93),)]
298
+
299
+
300
+ def _stress_rmsnorm(N=4097, M=37):
301
+ return [(_fixed((M, N), torch.float16, 64.0, 91), _fixed((N,), torch.float16, 1.0, 94)),
302
+ (_fixed((M, N), torch.bfloat16, 64.0, 92), _fixed((N,), torch.bfloat16, 1.0, 95)),
303
+ (_fixed((3, 8192), torch.float16, 32.0, 93), _fixed((8192,), torch.float16, 1.0, 96))]
304
+
305
+
306
+ def _stress_swiglu(N=4097, M=37):
307
+ return [(_fixed((M, N), torch.float16, 64.0, 91), _fixed((M, N), torch.float16, 8.0, 94)),
308
+ (_fixed((M, N), torch.bfloat16, 64.0, 92), _fixed((M, N), torch.bfloat16, 8.0, 95))]
309
+
310
+
311
+ def _stress_add_rmsnorm(N=4097, M=37):
312
+ return [(_fixed((M, N), torch.float16, 64.0, 91), _fixed((M, N), torch.float16, 64.0, 94),
313
+ _fixed((N,), torch.float16, 1.0, 97)),
314
+ (_fixed((M, N), torch.bfloat16, 64.0, 92), _fixed((M, N), torch.bfloat16, 64.0, 95),
315
+ _fixed((N,), torch.bfloat16, 1.0, 98))]
316
+
317
+
318
+ # A single FIXED, realistic LLM-regime shape so a candidate's speedup is COMPARABLE across
319
+ # the whole search (apples-to-apples), and the bench input is identical for kernel + both
320
+ # baselines. fp16, deterministic seed.
321
+ _BENCH_M, _BENCH_N = 8192, 4096
322
+
323
+
324
+ def _fixed(shape, dtype=torch.float16, scale=1.0, seed=1234, dev="cuda"):
325
+ g = torch.Generator(device=dev).manual_seed(seed)
326
+ return (torch.randn(shape, generator=g, device=dev, dtype=torch.float32) * scale).to(dtype)
327
+
328
+
329
+ def _bench_rmsnorm():
330
+ return (_fixed((_BENCH_M, _BENCH_N), seed=1), _fixed((_BENCH_N,), seed=2))
331
+
332
+
333
+ def _bench_softmax():
334
+ return (_fixed((_BENCH_M, _BENCH_N), seed=3),)
335
+
336
+
337
+ def _bench_swiglu():
338
+ return (_fixed((_BENCH_M, _BENCH_N), seed=4), _fixed((_BENCH_M, _BENCH_N), seed=5))
339
+
340
+
341
+ def _bench_add_rmsnorm():
342
+ return (_fixed((_BENCH_M, _BENCH_N), seed=6), _fixed((_BENCH_M, _BENCH_N), seed=7),
343
+ _fixed((_BENCH_N,), seed=8))
344
+
345
+
346
+ def _rope_costab(M, D, seed, dtype=torch.float16, dev="cuda"):
347
+ g = torch.Generator(device=dev).manual_seed(seed)
348
+ ang = torch.randn((M, D // 2), generator=g, device=dev, dtype=torch.float32)
349
+ c = torch.cos(ang); s = torch.sin(ang)
350
+ return torch.cat([c, c], -1).to(dtype), torch.cat([s, s], -1).to(dtype)
351
+
352
+
353
+ def _bench_rope():
354
+ M, D = 32768, 128
355
+ cos, sin = _rope_costab(M, D, 21)
356
+ return (_fixed((M, D), seed=20), cos, sin)
357
+
358
+
359
+ def _bench_layernorm():
360
+ return (_fixed((_BENCH_M, _BENCH_N), seed=30), _fixed((_BENCH_N,), seed=31), _fixed((_BENCH_N,), seed=32))
361
+
362
+
363
+ def _bench_add_layernorm():
364
+ return (_fixed((_BENCH_M, _BENCH_N), seed=33), _fixed((_BENCH_M, _BENCH_N), seed=34),
365
+ _fixed((_BENCH_N,), seed=35), _fixed((_BENCH_N,), seed=36))
366
+
367
+
368
+ def _bench_geglu():
369
+ return (_fixed((_BENCH_M, _BENCH_N), seed=37), _fixed((_BENCH_M, _BENCH_N), seed=38))
370
+
371
+
372
+ def _bench_qknorm_rope():
373
+ M, D = 32768, 128
374
+ cos, sin = _rope_costab(M, D, 41)
375
+ return (_fixed((M, D), seed=40), _fixed((D,), seed=42), cos, sin)
376
+
377
+
378
+ # ---- input generators (adversarial) ----------------------------------------------------
379
+ def _mk_rmsnorm(rng):
380
+ M, N, dt, sc = _pick(rng, _NROWS), _pick(rng, _ROWLEN), _pick(rng, _DTYPES), _pick(rng, _SCALES)
381
+ x = _randn(rng, (M, N), dt, sc)
382
+ w = _randn(rng, (N,), dt, 1.0)
383
+ return (x, w)
384
+
385
+
386
+ def _mk_softmax(rng):
387
+ M, N, dt, sc = _pick(rng, _NROWS), _pick(rng, _ROWLEN), _pick(rng, _DTYPES), _pick(rng, _SCALES)
388
+ return (_randn(rng, (M, N), dt, sc),)
389
+
390
+
391
+ def _mk_swiglu(rng):
392
+ M, N, dt, sc = _pick(rng, _NROWS), _pick(rng, _ROWLEN), _pick(rng, _DTYPES), _pick(rng, _SCALES)
393
+ return (_randn(rng, (M, N), dt, sc), _randn(rng, (M, N), dt, 1.0))
394
+
395
+
396
+ def _mk_add_rmsnorm(rng):
397
+ M, N, dt, sc = _pick(rng, _NROWS), _pick(rng, _ROWLEN), _pick(rng, _DTYPES), _pick(rng, _SCALES)
398
+ return (_randn(rng, (M, N), dt, sc), _randn(rng, (M, N), dt, sc), _randn(rng, (N,), dt, 1.0))
399
+
400
+
401
+ _HEADDIM = [64, 128, 256] # even head dims (RoPE)
402
+
403
+
404
+ def _mk_rope(rng):
405
+ M, D, dt, sc = _pick(rng, _NROWS), _pick(rng, _HEADDIM), _pick(rng, _DTYPES), _pick(rng, _SCALES)
406
+ g = torch.Generator(device="cuda").manual_seed(rng.randrange(2**31))
407
+ ang = torch.randn((M, D // 2), generator=g, device="cuda", dtype=torch.float32)
408
+ c = torch.cos(ang); s = torch.sin(ang)
409
+ cos = torch.cat([c, c], -1).to(dt); sin = torch.cat([s, s], -1).to(dt)
410
+ return (_randn(rng, (M, D), dt, sc), cos, sin)
411
+
412
+
413
+ def _stress_rope():
414
+ out = []
415
+ for D, dt in [(128, torch.float16), (256, torch.bfloat16), (64, torch.float16)]:
416
+ g = torch.Generator(device="cuda").manual_seed(700 + D)
417
+ ang = torch.randn((37, D // 2), generator=g, device="cuda", dtype=torch.float32)
418
+ c = torch.cos(ang); s = torch.sin(ang)
419
+ out.append((_fixed((37, D), dt, 64.0, 700 + D),
420
+ torch.cat([c, c], -1).to(dt), torch.cat([s, s], -1).to(dt)))
421
+ return out
422
+
423
+
424
+ def _mk_layernorm(rng):
425
+ M, N, dt, sc = _pick(rng, _NROWS), _pick(rng, _ROWLEN), _pick(rng, _DTYPES), _pick(rng, _SCALES)
426
+ return (_randn(rng, (M, N), dt, sc), _randn(rng, (N,), dt, 1.0), _randn(rng, (N,), dt, 1.0))
427
+
428
+
429
+ def _mk_add_layernorm(rng):
430
+ M, N, dt, sc = _pick(rng, _NROWS), _pick(rng, _ROWLEN), _pick(rng, _DTYPES), _pick(rng, _SCALES)
431
+ return (_randn(rng, (M, N), dt, sc), _randn(rng, (M, N), dt, sc),
432
+ _randn(rng, (N,), dt, 1.0), _randn(rng, (N,), dt, 1.0))
433
+
434
+
435
+ def _mk_geglu(rng):
436
+ M, N, dt, sc = _pick(rng, _NROWS), _pick(rng, _ROWLEN), _pick(rng, _DTYPES), _pick(rng, _SCALES)
437
+ return (_randn(rng, (M, N), dt, sc), _randn(rng, (M, N), dt, 1.0))
438
+
439
+
440
+ def _stress_layernorm():
441
+ return [(_fixed((37, 4097), torch.float16, 64.0, 91), _fixed((4097,), torch.float16, 1.0, 94),
442
+ _fixed((4097,), torch.float16, 1.0, 95)),
443
+ (_fixed((37, 4097), torch.bfloat16, 64.0, 92), _fixed((4097,), torch.bfloat16, 1.0, 96),
444
+ _fixed((4097,), torch.bfloat16, 1.0, 97))]
445
+
446
+
447
+ def _stress_add_layernorm():
448
+ return [(_fixed((37, 4097), torch.float16, 64.0, 91), _fixed((37, 4097), torch.float16, 64.0, 92),
449
+ _fixed((4097,), torch.float16, 1.0, 94), _fixed((4097,), torch.float16, 1.0, 95)),
450
+ (_fixed((37, 4097), torch.bfloat16, 64.0, 96), _fixed((37, 4097), torch.bfloat16, 64.0, 97),
451
+ _fixed((4097,), torch.bfloat16, 1.0, 98), _fixed((4097,), torch.bfloat16, 1.0, 99))]
452
+
453
+
454
+ def _stress_geglu():
455
+ return [(_fixed((37, 4097), torch.float16, 64.0, 91), _fixed((37, 4097), torch.float16, 8.0, 94)),
456
+ (_fixed((37, 4097), torch.bfloat16, 64.0, 92), _fixed((37, 4097), torch.bfloat16, 8.0, 95))]
457
+
458
+
459
+ def _mk_qknorm_rope(rng):
460
+ M, D, dt, sc = _pick(rng, _NROWS), _pick(rng, _HEADDIM), _pick(rng, _DTYPES), _pick(rng, _SCALES)
461
+ g = torch.Generator(device="cuda").manual_seed(rng.randrange(2**31))
462
+ ang = torch.randn((M, D // 2), generator=g, device="cuda", dtype=torch.float32)
463
+ c = torch.cos(ang); s = torch.sin(ang)
464
+ cos = torch.cat([c, c], -1).to(dt); sin = torch.cat([s, s], -1).to(dt)
465
+ return (_randn(rng, (M, D), dt, sc), _randn(rng, (D,), dt, 1.0), cos, sin)
466
+
467
+
468
+ def _stress_qknorm_rope():
469
+ out = []
470
+ for D, dt in [(128, torch.float16), (256, torch.bfloat16), (64, torch.float16)]:
471
+ g = torch.Generator(device="cuda").manual_seed(800 + D)
472
+ ang = torch.randn((37, D // 2), generator=g, device="cuda", dtype=torch.float32)
473
+ c = torch.cos(ang); s = torch.sin(ang)
474
+ out.append((_fixed((37, D), dt, 64.0, 800 + D), _fixed((D,), dt, 1.0, 810 + D),
475
+ torch.cat([c, c], -1).to(dt), torch.cat([s, s], -1).to(dt)))
476
+ return out
477
+
478
+
479
+ # ---------- comprehensive suite: input generators ----------------------------------------
480
+ def _mk_x(rng): # plain elementwise (M,N)
481
+ M, N, dt, sc = _pick(rng, _NROWS), _pick(rng, _ROWLEN), _pick(rng, _DTYPES), _pick(rng, _SCALES)
482
+ return (_randn(rng, (M, N), dt, sc),)
483
+
484
+
485
+ def _mk_x_bias(rng):
486
+ M, N, dt, sc = _pick(rng, _NROWS), _pick(rng, _ROWLEN), _pick(rng, _DTYPES), _pick(rng, _SCALES)
487
+ return (_randn(rng, (M, N), dt, sc), _randn(rng, (N,), dt, 1.0))
488
+
489
+
490
+ def _mk_gate_up(rng):
491
+ M, N, dt, sc = _pick(rng, _NROWS), _pick(rng, _ROWLEN), _pick(rng, _DTYPES), _pick(rng, _SCALES)
492
+ return (_randn(rng, (M, N), dt, sc), _randn(rng, (M, N), dt, 1.0))
493
+
494
+
495
+ def _mk_softmax_scale(rng):
496
+ M, N, dt, sc = _pick(rng, _NROWS), _pick(rng, _ROWLEN), _pick(rng, _DTYPES), _pick(rng, _SCALES)
497
+ scl = torch.tensor([0.05 + rng.random() * 1.5], device="cuda", dtype=dt)
498
+ return (_randn(rng, (M, N), dt, sc), scl)
499
+
500
+
501
+ def _mk_layernorm_gelu(rng):
502
+ M, N, dt, sc = _pick(rng, _NROWS), _pick(rng, _ROWLEN), _pick(rng, _DTYPES), _pick(rng, _SCALES)
503
+ return (_randn(rng, (M, N), dt, sc), _randn(rng, (N,), dt, 1.0), _randn(rng, (N,), dt, 1.0))
504
+
505
+
506
+ def _mk_add_rmsnorm_rope(rng):
507
+ M, D, dt, sc = _pick(rng, _NROWS), _pick(rng, _HEADDIM), _pick(rng, _DTYPES), _pick(rng, _SCALES)
508
+ g = torch.Generator(device="cuda").manual_seed(rng.randrange(2**31))
509
+ ang = torch.randn((M, D // 2), generator=g, device="cuda", dtype=torch.float32)
510
+ c = torch.cos(ang); s = torch.sin(ang)
511
+ return (_randn(rng, (M, D), dt, sc), _randn(rng, (M, D), dt, sc), _randn(rng, (D,), dt, 1.0),
512
+ torch.cat([c, c], -1).to(dt), torch.cat([s, s], -1).to(dt))
513
+
514
+
515
+ def _int8(rng, shape, dev="cuda"):
516
+ g = torch.Generator(device=dev).manual_seed(rng.randrange(2**31))
517
+ return torch.randint(-127, 127, shape, generator=g, device=dev, dtype=torch.int8)
518
+
519
+
520
+ def _mk_dequant_int8(rng):
521
+ M, N = _pick(rng, _NROWS), _pick(rng, _ROWLEN)
522
+ q = _int8(rng, (M, N))
523
+ g = torch.Generator(device="cuda").manual_seed(rng.randrange(2**31))
524
+ scale = (torch.rand((M, 1), generator=g, device="cuda") * 0.05 + 0.005).to(torch.float16)
525
+ return (q, scale)
526
+
527
+
528
+ # bench inputs (fixed large)
529
+ def _bench_x(seed): return (_fixed((_BENCH_M, _BENCH_N), seed=seed),)
530
+ def _bench_gelu(): return _bench_x(50)
531
+ def _bench_silu(): return _bench_x(51)
532
+ def _bench_relu2(): return _bench_x(52)
533
+ def _bench_bias_gelu(): return (_fixed((_BENCH_M, _BENCH_N), seed=53), _fixed((_BENCH_N,), seed=54))
534
+ def _bench_reglu(): return (_fixed((_BENCH_M, _BENCH_N), seed=55), _fixed((_BENCH_M, _BENCH_N), seed=56))
535
+ def _bench_l2norm(): return _bench_x(57)
536
+ def _bench_log_softmax(): return _bench_x(58)
537
+ def _bench_softmax_scale(): return (_fixed((_BENCH_M, _BENCH_N), seed=59),
538
+ torch.tensor([0.125], device="cuda", dtype=torch.float16))
539
+ def _bench_layernorm_gelu(): return (_fixed((_BENCH_M, _BENCH_N), seed=60),
540
+ _fixed((_BENCH_N,), seed=61), _fixed((_BENCH_N,), seed=62))
541
+ def _bench_add_rmsnorm_rope():
542
+ M, D = 32768, 128; cos, sin = _rope_costab(M, D, 63)
543
+ return (_fixed((M, D), seed=64), _fixed((M, D), seed=65), _fixed((D,), seed=66), cos, sin)
544
+ _TALL_M = [4, 8, 16, 32]
545
+ _WIDE_N = [16384, 32768, 65536, 131072]
546
+ def _mk_rmsnorm_wide(rng):
547
+ M, N, dt, sc = _pick(rng, _TALL_M), _pick(rng, _WIDE_N), _pick(rng, _DTYPES), _pick(rng, _SCALES)
548
+ return (_randn(rng, (M, N), dt, sc), _randn(rng, (N,), dt, 1.0))
549
+ def _bench_rmsnorm_wide(): return (_fixed((8, 131072), seed=90), _fixed((131072,), seed=91))
550
+ def _stress_rmsnorm_wide():
551
+ return [(_fixed((8, 131072), torch.float16, 64.0, 90), _fixed((131072,), torch.float16, 1.0, 91)),
552
+ (_fixed((5, 65537), torch.bfloat16, 32.0, 92), _fixed((65537,), torch.bfloat16, 1.0, 93)),
553
+ (_fixed((16, 40000), torch.float16, 16.0, 94), _fixed((40000,), torch.float16, 1.0, 95))]
554
+ def _bench_dequant_int8():
555
+ g = torch.Generator(device="cuda").manual_seed(67)
556
+ q = torch.randint(-127, 127, (_BENCH_M, _BENCH_N), generator=g, device="cuda", dtype=torch.int8)
557
+ g2 = torch.Generator(device="cuda").manual_seed(68)
558
+ scale = (torch.rand((_BENCH_M, 1), generator=g2, device="cuda") * 0.05 + 0.005).to(torch.float16)
559
+ return (q, scale)
560
+
561
+
562
+ # stress (guaranteed hard) — high magnitude / low precision / odd N
563
+ def _stress_x():
564
+ return [(_fixed((37, 4097), torch.float16, 64.0, 71),), (_fixed((37, 4097), torch.bfloat16, 64.0, 72),),
565
+ (_fixed((3, 8192), torch.float16, 32.0, 73),)]
566
+ def _stress_x_bias():
567
+ return [(_fixed((37, 4097), torch.float16, 64.0, 71), _fixed((4097,), torch.float16, 4.0, 74)),
568
+ (_fixed((37, 4097), torch.bfloat16, 64.0, 72), _fixed((4097,), torch.bfloat16, 4.0, 75))]
569
+ def _stress_gate_up():
570
+ return [(_fixed((37, 4097), torch.float16, 64.0, 71), _fixed((37, 4097), torch.float16, 8.0, 74)),
571
+ (_fixed((37, 4097), torch.bfloat16, 64.0, 72), _fixed((37, 4097), torch.bfloat16, 8.0, 75))]
572
+ def _stress_softmax_scale():
573
+ return [(_fixed((37, 4097), torch.float16, 64.0, 71), torch.tensor([1.5], device="cuda", dtype=torch.float16)),
574
+ (_fixed((37, 4097), torch.bfloat16, 32.0, 72), torch.tensor([0.5], device="cuda", dtype=torch.bfloat16))]
575
+ def _stress_layernorm_gelu():
576
+ return [(_fixed((37, 4097), torch.float16, 64.0, 71), _fixed((4097,), torch.float16, 1.0, 74),
577
+ _fixed((4097,), torch.float16, 1.0, 75)),
578
+ (_fixed((37, 4097), torch.bfloat16, 64.0, 72), _fixed((4097,), torch.bfloat16, 1.0, 76),
579
+ _fixed((4097,), torch.bfloat16, 1.0, 77))]
580
+ def _stress_add_rmsnorm_rope():
581
+ out = []
582
+ for D, dt in [(128, torch.float16), (256, torch.bfloat16)]:
583
+ g = torch.Generator(device="cuda").manual_seed(820 + D)
584
+ ang = torch.randn((37, D // 2), generator=g, device="cuda", dtype=torch.float32)
585
+ c = torch.cos(ang); s = torch.sin(ang)
586
+ out.append((_fixed((37, D), dt, 64.0, 820 + D), _fixed((37, D), dt, 64.0, 830 + D),
587
+ _fixed((D,), dt, 1.0, 840 + D), torch.cat([c, c], -1).to(dt), torch.cat([s, s], -1).to(dt)))
588
+ return out
589
+ def _stress_dequant_int8():
590
+ out = []
591
+ for M, N, sd in [(37, 4097, 71), (3, 8192, 72)]:
592
+ g = torch.Generator(device="cuda").manual_seed(sd)
593
+ q = torch.randint(-127, 127, (M, N), generator=g, device="cuda", dtype=torch.int8)
594
+ g2 = torch.Generator(device="cuda").manual_seed(sd + 1)
595
+ scale = (torch.rand((M, 1), generator=g2, device="cuda") * 0.1 + 0.005).to(torch.float16)
596
+ out.append((q, scale))
597
+ return out
598
+
599
+
600
+ # ---- V2 standalone ops: generators / bench / stress --------------------------------------
601
+ def _mk_rope_inter(rng):
602
+ M, D, dt, sc = _pick(rng, _NROWS), _pick(rng, _HEADDIM), _pick(rng, _DTYPES), _pick(rng, _SCALES)
603
+ g = torch.Generator(device="cuda").manual_seed(rng.randrange(2**31))
604
+ ang = torch.randn((M, D // 2), generator=g, device="cuda", dtype=torch.float32)
605
+ return (_randn(rng, (M, D), dt, sc), torch.cos(ang).to(dt), torch.sin(ang).to(dt))
606
+
607
+
608
+ def _bench_rope_inter():
609
+ M, D = 32768, 128
610
+ g = torch.Generator(device="cuda").manual_seed(140)
611
+ ang = torch.randn((M, D // 2), generator=g, device="cuda", dtype=torch.float32)
612
+ return (_fixed((M, D), seed=141), torch.cos(ang).to(torch.float16), torch.sin(ang).to(torch.float16))
613
+
614
+
615
+ def _stress_rope_inter():
616
+ out = []
617
+ for D, dt in [(128, torch.float16), (256, torch.bfloat16), (64, torch.float16)]:
618
+ g = torch.Generator(device="cuda").manual_seed(850 + D)
619
+ ang = torch.randn((37, D // 2), generator=g, device="cuda", dtype=torch.float32)
620
+ out.append((_fixed((37, D), dt, 64.0, 850 + D), torch.cos(ang).to(dt), torch.sin(ang).to(dt)))
621
+ return out
622
+
623
+
624
+ def _tgt(M, N, seed):
625
+ g = torch.Generator(device="cuda").manual_seed(seed)
626
+ return torch.randint(0, N, (M,), generator=g, device="cuda", dtype=torch.int64)
627
+
628
+
629
+ def _mk_cross_entropy(rng):
630
+ M, N, dt, sc = _pick(rng, _NROWS), _pick(rng, _ROWLEN), _pick(rng, _DTYPES), _pick(rng, _SCALES)
631
+ return (_randn(rng, (M, N), dt, sc), _tgt(M, N, rng.randrange(2**31)))
632
+
633
+
634
+ def _bench_cross_entropy():
635
+ return (_fixed((_BENCH_M, _BENCH_N), seed=150), _tgt(_BENCH_M, _BENCH_N, 151))
636
+
637
+
638
+ def _stress_cross_entropy():
639
+ return [(_fixed((37, 4097), torch.float16, 64.0, 152), _tgt(37, 4097, 153)),
640
+ (_fixed((37, 4097), torch.bfloat16, 64.0, 154), _tgt(37, 4097, 155)),
641
+ (_fixed((3, 8192), torch.float16, 32.0, 156), _tgt(3, 8192, 157))]
642
+
643
+
644
+ def _bench_glu(): return (_fixed((_BENCH_M, _BENCH_N), seed=160), _fixed((_BENCH_M, _BENCH_N), seed=161))
645
+ def _bench_softcap_softmax(): return _bench_x(162)
646
+ def _bench_rmsnorm_gemma(): return (_fixed((_BENCH_M, _BENCH_N), seed=163), _fixed((_BENCH_N,), seed=164))
647
+
648
+
649
+ # ---- invention suite: generators / bench / stress ----------------------------------------
650
+ def _mk_xy(rng):
651
+ M, N, dt, sc = _pick(rng, _NROWS), _pick(rng, _ROWLEN), _pick(rng, _DTYPES), _pick(rng, _SCALES)
652
+ return (_randn(rng, (M, N), dt, sc), _randn(rng, (M, N), dt, sc))
653
+
654
+
655
+ def _stress_xy():
656
+ return [(_fixed((37, 4097), torch.float16, 64.0, 171), _fixed((37, 4097), torch.float16, 64.0, 172)),
657
+ (_fixed((37, 4097), torch.bfloat16, 64.0, 173), _fixed((37, 4097), torch.bfloat16, 64.0, 174)),
658
+ (_fixed((3, 8192), torch.float16, 32.0, 175), _fixed((3, 8192), torch.float16, 32.0, 176))]
659
+
660
+
661
+ def _bench_cumsum(): return _bench_x(170)
662
+ def _bench_entropy(): return _bench_x(177)
663
+ def _bench_kl_div(): return (_fixed((_BENCH_M, _BENCH_N), seed=178), _fixed((_BENCH_M, _BENCH_N), seed=179))
664
+
665
+
666
+ SPECS: dict[str, OpSpec] = {
667
+ "rmsnorm": OpSpec(
668
+ name="rmsnorm",
669
+ reference=_rmsnorm_ref,
670
+ make_inputs=_mk_rmsnorm,
671
+ bench_inputs=_bench_rmsnorm,
672
+ stress_inputs=_stress_rmsnorm,
673
+ signature_hint=(
674
+ "def run(x, w): # x:(M,N) w:(N,) -> (M,N)\n"
675
+ " # y[i] = x[i] * rsqrt(mean(x[i]^2) + 1e-6) * w (RMSNorm, accumulate in fp32)"),
676
+ notes="row-wise normalise + per-channel scale; eager does square/mean/rsqrt/mul as separate launches.",
677
+ extra={"eps": 1e-6},
678
+ ),
679
+ "softmax": OpSpec(
680
+ name="softmax",
681
+ reference=_softmax_ref,
682
+ make_inputs=_mk_softmax,
683
+ bench_inputs=_bench_softmax,
684
+ stress_inputs=_stress_1tensor,
685
+ signature_hint=(
686
+ "def run(x): # x:(M,N) -> (M,N)\n"
687
+ " # row-wise softmax. MUST subtract row-max before exp (numerical stability)."),
688
+ notes="classic online/row softmax; the max-subtraction is the correctness trap large scales expose.",
689
+ ),
690
+ "swiglu": OpSpec(
691
+ name="swiglu",
692
+ reference=_swiglu_ref,
693
+ make_inputs=_mk_swiglu,
694
+ bench_inputs=_bench_swiglu,
695
+ stress_inputs=_stress_swiglu,
696
+ signature_hint=(
697
+ "def run(gate, up): # both (M,N) -> (M,N)\n"
698
+ " # SiLU(gate) * up where SiLU(z) = z * sigmoid(z) (FFN gate fusion)"),
699
+ notes="elementwise activation fusion; eager launches silu + mul separately.",
700
+ ),
701
+ "add_rmsnorm": OpSpec(
702
+ name="add_rmsnorm",
703
+ reference=_add_rmsnorm_ref,
704
+ make_inputs=_mk_add_rmsnorm,
705
+ bench_inputs=_bench_add_rmsnorm,
706
+ stress_inputs=_stress_add_rmsnorm,
707
+ signature_hint=(
708
+ "def run(x, residual, w): # x,residual:(M,N) w:(N,) -> (M,N)\n"
709
+ " # h = x + residual; then RMSNorm(h) * w (fused residual-add + norm)"),
710
+ notes="the transformer block's add-then-norm; eager does add then a separate norm pass.",
711
+ extra={"eps": 1e-6},
712
+ ),
713
+ "rope": OpSpec(
714
+ name="rope",
715
+ reference=_rope_ref,
716
+ make_inputs=_mk_rope,
717
+ bench_inputs=_bench_rope,
718
+ stress_inputs=_stress_rope,
719
+ signature_hint=(
720
+ "def run(x, cos, sin): # x,cos,sin all (M,D), D even -> (M,D)\n"
721
+ " # LLaMA rotate_half RoPE: out = x*cos + rotate_half(x)*sin,\n"
722
+ " # rotate_half(x) = concat(-x[:, D/2:], x[:, :D/2]). Accumulate in fp32."),
723
+ notes="fused rotary embedding; eager does a cat (alloc) + 2 muls + add as separate launches.",
724
+ ),
725
+ "layernorm": OpSpec(
726
+ name="layernorm",
727
+ reference=_layernorm_ref,
728
+ make_inputs=_mk_layernorm,
729
+ bench_inputs=_bench_layernorm,
730
+ stress_inputs=_stress_layernorm,
731
+ signature_hint=(
732
+ "def run(x, w, b): # x:(M,N) w,b:(N,) -> (M,N)\n"
733
+ " # LayerNorm: y = (x-mean)/sqrt(var+1e-5)*w + b, over the last dim. fp32 reductions."),
734
+ notes="full layernorm (mean+var+affine); eager launches mean/var/normalize/affine separately.",
735
+ extra={"eps": 1e-5},
736
+ ),
737
+ "add_layernorm": OpSpec(
738
+ name="add_layernorm",
739
+ reference=_add_layernorm_ref,
740
+ make_inputs=_mk_add_layernorm,
741
+ bench_inputs=_bench_add_layernorm,
742
+ stress_inputs=_stress_add_layernorm,
743
+ signature_hint=(
744
+ "def run(x, residual, w, b): # x,residual:(M,N) w,b:(N,) -> (M,N)\n"
745
+ " # h = x + residual; then LayerNorm(h)*w + b (fused residual-add + layernorm)"),
746
+ notes="residual-add + full layernorm; the transformer block's add-then-LN.",
747
+ extra={"eps": 1e-5},
748
+ ),
749
+ "geglu": OpSpec(
750
+ name="geglu",
751
+ reference=_geglu_ref,
752
+ make_inputs=_mk_geglu,
753
+ bench_inputs=_bench_geglu,
754
+ stress_inputs=_stress_geglu,
755
+ signature_hint=(
756
+ "def run(gate, up): # both (M,N) -> (M,N)\n"
757
+ " # GeGLU: gelu_tanh(gate) * up, gelu_tanh(z)=0.5*z*(1+tanh(0.7978845608*(z+0.044715*z^3)))"),
758
+ notes="GELU-gated FFN fusion (tanh-approx GELU); eager launches gelu then mul separately.",
759
+ ),
760
+ "qknorm_rope": OpSpec(
761
+ name="qknorm_rope",
762
+ reference=_qknorm_rope_ref,
763
+ make_inputs=_mk_qknorm_rope,
764
+ bench_inputs=_bench_qknorm_rope,
765
+ stress_inputs=_stress_qknorm_rope,
766
+ signature_hint=(
767
+ "def run(x, w, cos, sin): # x,cos,sin:(M,D) w:(D,), D even -> (M,D)\n"
768
+ " # FUSED QK-norm+RoPE: n = rmsnorm(x)*w (over D); then rope(n): \n"
769
+ " # out = n*cos + rotate_half(n)*sin. Keep the rms scale in-register (no\n"
770
+ " # intermediate n in DRAM). Accumulate the reduction in fp32."),
771
+ notes="reduction->gather fusion chain (Qwen-style QK-norm then RoPE); the bigger-discovery probe.",
772
+ extra={"eps": 1e-6},
773
+ ),
774
+ # ---- comprehensive suite -------------------------------------------------------------
775
+ "gelu": OpSpec("gelu", _gelu_ref, _mk_x, _bench_gelu, _stress_x,
776
+ "def run(x): # (M,N)->(M,N) gelu_tanh(x)=0.5*x*(1+tanh(0.7978845608*(x+0.044715*x^3)))",
777
+ notes="GELU activation (tanh approx)."),
778
+ "silu": OpSpec("silu", _silu_ref, _mk_x, _bench_silu, _stress_x,
779
+ "def run(x): # (M,N)->(M,N) silu(x)=x*sigmoid(x)", notes="SiLU/Swish activation."),
780
+ "relu2": OpSpec("relu2", _relu2_ref, _mk_x, _bench_relu2, _stress_x,
781
+ "def run(x): # (M,N)->(M,N) relu(x)^2", notes="squared-ReLU activation (used in some FFNs)."),
782
+ "bias_gelu": OpSpec("bias_gelu", _bias_gelu_ref, _mk_x_bias, _bench_bias_gelu, _stress_x_bias,
783
+ "def run(x, bias): # x:(M,N) bias:(N,) -> (M,N) gelu_tanh(x + bias)",
784
+ notes="fused bias-add + GELU (the FFN up-proj epilogue)."),
785
+ "reglu": OpSpec("reglu", _reglu_ref, _mk_gate_up, _bench_reglu, _stress_gate_up,
786
+ "def run(gate, up): # both (M,N) -> (M,N) relu(gate) * up",
787
+ notes="ReGLU gated FFN fusion."),
788
+ "l2norm": OpSpec("l2norm", _l2norm_ref, _mk_x, _bench_l2norm, _stress_x,
789
+ "def run(x): # (M,N)->(M,N) x * rsqrt(sum(x^2) + 1e-6) (L2 normalize over last dim)",
790
+ notes="L2 normalization (reduction); fp32 accumulate.", extra={"eps": 1e-6}),
791
+ "log_softmax": OpSpec("log_softmax", _log_softmax_ref, _mk_x, _bench_log_softmax, _stress_x,
792
+ "def run(x): # (M,N)->(M,N) x - logsumexp(x); MUST subtract row-max for stability",
793
+ notes="row-wise log-softmax (reduction); max-subtraction required."),
794
+ "softmax_scale": OpSpec("softmax_scale", _softmax_scale_ref, _mk_softmax_scale, _bench_softmax_scale,
795
+ _stress_softmax_scale,
796
+ "def run(x, scale): # x:(M,N) scale:(1,) -> (M,N) softmax(x*scale[0]); subtract row-max",
797
+ notes="fused scale + softmax (attention prelude); reduction."),
798
+ "layernorm_gelu": OpSpec("layernorm_gelu", _layernorm_gelu_ref, _mk_layernorm_gelu, _bench_layernorm_gelu,
799
+ _stress_layernorm_gelu,
800
+ "def run(x, w, b): # x:(M,N) w,b:(N,) -> (M,N) gelu(LayerNorm(x)*w + b)",
801
+ notes="FUSION CHAIN: layernorm (reduction) -> GELU epilogue.", extra={"eps": 1e-5}),
802
+ "add_rmsnorm_rope": OpSpec("add_rmsnorm_rope", _add_rmsnorm_rope_ref, _mk_add_rmsnorm_rope,
803
+ _bench_add_rmsnorm_rope, _stress_add_rmsnorm_rope,
804
+ "def run(x, residual, w, cos, sin): # x,res,cos,sin:(M,D) w:(D,) -> (M,D)\n"
805
+ " # h=x+residual; n=rmsnorm(h)*w; out=n*cos+rotate_half(n)*sin (3-stage fusion chain)",
806
+ notes="THREE-stage fusion chain: residual-add -> RMSNorm (reduction) -> RoPE (gather).",
807
+ extra={"eps": 1e-6}),
808
+ "dequant_int8": OpSpec("dequant_int8", _dequant_int8_ref, _mk_dequant_int8, _bench_dequant_int8,
809
+ _stress_dequant_int8,
810
+ "def run(q, scale): # q:(M,N) int8, scale:(M,1) fp16 -> (M,N) fp16 out = q*scale (per-row dequant)",
811
+ notes="NON-GEMM int8 weight dequantization (memory-bound)."),
812
+ # ---- V2 standalone ops --------------------------------------------------------------
813
+ "softcap_softmax": OpSpec("softcap_softmax", _softcap_softmax_ref, _mk_softmax,
814
+ _bench_softcap_softmax, _stress_1tensor,
815
+ "def run(x): # (M,N)->(M,N) softmax(30*tanh(x/30)) row-wise (Gemma2 logit softcap).\n"
816
+ " # MUST apply the softcap BEFORE softmax and subtract the row-max before exp.",
817
+ notes="Gemma2-style softcapped softmax; the cap is what large-scale inputs expose."),
818
+ "rmsnorm_gemma": OpSpec("rmsnorm_gemma", _rmsnorm_gemma_ref, _mk_rmsnorm,
819
+ _bench_rmsnorm_gemma, _stress_rmsnorm,
820
+ "def run(x, w): # x:(M,N) w:(N,) -> (M,N)\n"
821
+ " # Gemma RMSNorm: y = x * rsqrt(mean(x^2)+1e-6) * (1 + w) — note the (1 + w)!",
822
+ notes="Gemma-style RMSNorm: scale by (1+w); dropping the +1 is the classic silent bug.",
823
+ extra={"eps": 1e-6}),
824
+ "glu": OpSpec("glu", _glu_ref, _mk_gate_up, _bench_glu, _stress_gate_up,
825
+ "def run(gate, up): # both (M,N) -> (M,N) sigmoid(gate) * up (the original GLU)",
826
+ notes="the original gated linear unit; eager launches sigmoid then mul separately."),
827
+ "rope_interleaved": OpSpec("rope_interleaved", _rope_interleaved_ref, _mk_rope_inter,
828
+ _bench_rope_inter, _stress_rope_inter,
829
+ "def run(x, cos, sin): # x:(M,D), cos,sin:(M,D/2), D even -> (M,D)\n"
830
+ " # GPT-J INTERLEAVED RoPE: out[2i]=x[2i]*cos[i]-x[2i+1]*sin[i];\n"
831
+ " # out[2i+1]=x[2i+1]*cos[i]+x[2i]*sin[i]. fp32 math.",
832
+ notes="interleaved-pair rotary (GPT-J/NeoX-style); strided pair access is the fusion win."),
833
+ "cross_entropy": OpSpec("cross_entropy", _cross_entropy_ref, _mk_cross_entropy,
834
+ _bench_cross_entropy, _stress_cross_entropy,
835
+ "def run(x, tgt): # x:(M,N) fp, tgt:(M,) int64 -> (M,)\n"
836
+ " # per-row cross-entropy: logsumexp(x) - x[tgt]. MUST subtract row-max inside\n"
837
+ " # the logsumexp (stability). Accumulate in fp32; output dtype = x.dtype.",
838
+ notes="fused cross-entropy (the Liger flagship): softmax+log+gather in one pass."),
839
+ # ---- INVENTION suite: never-trained problem classes ----------------------------------
840
+ "cumsum": OpSpec("cumsum", _cumsum_ref, _mk_x, _bench_cumsum, _stress_x,
841
+ "def run(x): # (M,N)->(M,N) row-wise INCLUSIVE prefix sum (cumsum along the last dim).\n"
842
+ " # This is a SCAN: each output depends on ALL previous elements in the row —\n"
843
+ " # a carry must propagate across blocks. Accumulate in fp32.",
844
+ notes="prefix-scan algorithm class (carry across blocks) — unlike every reduction op.",
845
+ # SCAN tolerance: rounding error tracks the RUNNING-SUM magnitude (scale*sqrt(j)),
846
+ # not the output element — fp32 envelope: eps*scale_max*sqrt(N_max) ≈ 1.2e-7*64*90
847
+ # ≈ 7e-4, ×~constants → 5e-3 atol. Wrong-kernel errors here are 1e3+, so the
848
+ # verification stays sharp. fp16/bf16 global atol already dominates this term.
849
+ tol_override={torch.float32: (2e-4, 5e-3)}),
850
+ "entropy": OpSpec("entropy", _entropy_ref, _mk_x, _bench_entropy, _stress_x,
851
+ "def run(x): # x:(M,N) -> (M,) Shannon entropy of softmax(x) per row:\n"
852
+ " # H = logsumexp(x) - sum(x * softmax(x)). MUST subtract the row max inside\n"
853
+ " # both the logsumexp and the softmax (stability). fp32 accumulation.",
854
+ notes="fused entropy-from-logits (sampling diagnostics): two coupled reductions.",
855
+ # H = lse - Σx·p subtracts two O(scale·|x|) quantities — the REFERENCE itself
856
+ # carries this cancellation, so elementwise agreement beyond eps*|x|_max*sqrt(N)
857
+ # ≈ 1.2e-7*300*64 ≈ 2e-3 is unattainable for ANY correct kernel. fp32 atol 2e-2
858
+ # against H ∈ [0, log N≈9]; the no-max-sub control still fails with nan/inf.
859
+ tol_override={torch.float32: (2e-4, 2e-2)}),
860
+ "kl_div": OpSpec("kl_div", _kl_div_ref, _mk_xy, _bench_kl_div, _stress_xy,
861
+ "def run(x, y): # both (M,N) logits -> (M,) KL(softmax(x) || softmax(y)) per row:\n"
862
+ " # lx = x - lse(x); ly = y - lse(y); out = sum(exp(lx) * (lx - ly)).\n"
863
+ " # BOTH logsumexps must be max-subtracted. fp32 accumulation.",
864
+ notes="fused distillation KL from raw logit pairs: double logsumexp + weighted sum."),
865
+ # ---- algorithm-discovery experiment: rmsnorm at a TALL-SKINNY shape where split-K wins ---
866
+ "rmsnorm_wide": OpSpec("rmsnorm_wide", _rmsnorm_ref, _mk_rmsnorm_wide, _bench_rmsnorm_wide,
867
+ _stress_rmsnorm_wide,
868
+ "def run(x, w): # x:(M,N) w:(N,) -> (M,N) RMSNorm; M is SMALL, N is HUGE (tall-skinny).\n"
869
+ " # one-program-per-row leaves the GPU idle; split the row's reduction across programs.",
870
+ notes="rmsnorm at tall-skinny (M<<#SMs, huge N) — the regime where split-K reduction wins.",
871
+ extra={"eps": 1e-6}),
872
+ }
873
+
874
+
875
+ def _build_chain_spec(norm: str, residual: bool, acts: list[str]) -> "OpSpec":
876
+ """Dynamic spec for ANY block stack a user assembles: [+residual] -> {rms|layer}norm ->
877
+ act1 -> act2 -> ... The reference is composed from the SAME blocks, so any stack the user
878
+ invents is verifiable (that is the whole point — the blocks ARE the spec)."""
879
+ import chains
880
+ eps = 1e-6 if norm == "rms" else 1e-5
881
+ fns = [chains.ACTS[a][0] for a in acts] # torch fns, in order
882
+
883
+ def ref(*args):
884
+ if residual and norm == "rms":
885
+ x, r, w = args; h = x.float() + r.float(); b = None
886
+ elif residual:
887
+ x, r, w, b = args; h = x.float() + r.float()
888
+ elif norm == "rms":
889
+ x, w = args; h = x.float(); b = None
890
+ else:
891
+ x, w, b = args; h = x.float()
892
+ if norm == "rms":
893
+ n = h * torch.rsqrt(h.pow(2).mean(-1, keepdim=True) + eps) * w.float()
894
+ else:
895
+ mu = h.mean(-1, keepdim=True); hc = h - mu
896
+ n = hc * torch.rsqrt((hc * hc).mean(-1, keepdim=True) + eps) * w.float() + b.float()
897
+ for fn in fns:
898
+ n = fn(n)
899
+ return n.to(args[0].dtype)
900
+
901
+ kind = ("add_" if residual else "") + ("rms" if norm == "rms" else "ln")
902
+ mk = {"rms": _mk_rmsnorm, "add_rms": _mk_add_rmsnorm, "ln": _mk_layernorm,
903
+ "add_ln": _mk_add_layernorm}[kind]
904
+ bench = {"rms": _bench_rmsnorm, "add_rms": _bench_add_rmsnorm, "ln": _bench_layernorm,
905
+ "add_ln": _bench_add_layernorm}[kind]
906
+ stress = {"rms": _stress_rmsnorm, "add_rms": _stress_add_rmsnorm, "ln": _stress_layernorm,
907
+ "add_ln": _stress_add_layernorm}[kind]
908
+ sig = {"rms": "def run(x, w):", "add_rms": "def run(x, residual, w):",
909
+ "ln": "def run(x, w, b):", "add_ln": "def run(x, residual, w, b):"}[kind]
910
+ normname = "RMSNorm" if norm == "rms" else "LayerNorm"
911
+ chain_str = " then ".join(acts)
912
+ return OpSpec("custom_chain", ref, mk, bench, stress,
913
+ f"{sig} # fused custom chain: {'residual + ' if residual else ''}{normname} "
914
+ f"then {chain_str}; accumulate the reduction in fp32",
915
+ notes=f"custom block stack: {'+res ' if residual else ''}{norm}norm -> {chain_str}")
916
+
917
+
918
+ def get_spec(name: str) -> OpSpec:
919
+ # dynamic block-stack spec, encoded as "chain|<norm>|<0|1>|<act1,act2,...>"
920
+ if name.startswith("chain|"):
921
+ _, norm, res, acts = name.split("|", 3)
922
+ return _build_chain_spec(norm, res == "1", [a for a in acts.split(",") if a])
923
+ if name not in SPECS:
924
+ raise KeyError(f"unknown op {name!r}; have {sorted(SPECS)}")
925
+ return SPECS[name]
926
+
927
+
928
+ # ----------------------------------------------------------------------------------------
929
+ # SHAPE-GRID inputs (V2, purely ADDITIVE — references/tolerances/bench_inputs untouched).
930
+ # Builds inputs for any op at an arbitrary (M, N[, dtype]) so the harness can re-bench the
931
+ # same kernel across a grid of shapes. For rope-family ops N is the head dim D (must be
932
+ # even). Deterministic seeds derived from the shape so every grid cell is reproducible.
933
+ # ----------------------------------------------------------------------------------------
934
+ _ROPE_FAMILY = {"rope", "rope_interleaved", "qknorm_rope", "add_rmsnorm_rope"}
935
+
936
+
937
+ def _grid_kind(name: str) -> str:
938
+ """Input-signature kind for an op (mirrors _register_chains's _IN map + explicit ops)."""
939
+ explicit = {
940
+ "softmax": "x", "log_softmax": "x", "gelu": "x", "silu": "x", "relu2": "x",
941
+ "l2norm": "x", "softcap_softmax": "x",
942
+ "rmsnorm": "rms", "rmsnorm_wide": "rms", "rmsnorm_gemma": "rms",
943
+ "layernorm": "ln", "layernorm_gelu": "ln",
944
+ "add_rmsnorm": "add_rms", "add_layernorm": "add_ln",
945
+ "swiglu": "gate_up", "geglu": "gate_up", "reglu": "gate_up", "glu": "gate_up",
946
+ "bias_gelu": "x_bias", "softmax_scale": "x_scale", "dequant_int8": "int8",
947
+ "cross_entropy": "ce", "rope": "rope", "rope_interleaved": "rope_inter",
948
+ "qknorm_rope": "qkr", "add_rmsnorm_rope": "arr",
949
+ "cumsum": "x", "entropy": "x", "kl_div": "gate_up",
950
+ }
951
+ if name in explicit:
952
+ return explicit[name]
953
+ if name.endswith("_short"):
954
+ return _grid_kind(name[: -len("_short")])
955
+ try:
956
+ import chains
957
+ for cname, kind, _ref, _s in chains.all_chains():
958
+ if cname == name:
959
+ return {"rms": "rms", "add_rms": "add_rms", "ln": "ln", "add_ln": "add_ln"}[kind]
960
+ except Exception:
961
+ pass
962
+ raise KeyError(f"no grid input builder for op {name!r}")
963
+
964
+
965
+ def grid_inputs(name: str, M: int, N: int, dtype=torch.float16) -> tuple:
966
+ kind = _grid_kind(name)
967
+ sd = (hash((name, M, N, str(dtype))) & 0x7FFFFFF) + 7
968
+ x = _fixed((M, N), dtype, 1.0, seed=sd)
969
+ if kind == "x":
970
+ return (x,)
971
+ if kind == "rms":
972
+ return (x, _fixed((N,), dtype, 1.0, seed=sd + 1))
973
+ if kind == "ln":
974
+ return (x, _fixed((N,), dtype, 1.0, seed=sd + 1), _fixed((N,), dtype, 1.0, seed=sd + 2))
975
+ if kind == "add_rms":
976
+ return (x, _fixed((M, N), dtype, 1.0, seed=sd + 3), _fixed((N,), dtype, 1.0, seed=sd + 1))
977
+ if kind == "add_ln":
978
+ return (x, _fixed((M, N), dtype, 1.0, seed=sd + 3),
979
+ _fixed((N,), dtype, 1.0, seed=sd + 1), _fixed((N,), dtype, 1.0, seed=sd + 2))
980
+ if kind == "gate_up":
981
+ return (x, _fixed((M, N), dtype, 1.0, seed=sd + 3))
982
+ if kind == "x_bias":
983
+ return (x, _fixed((N,), dtype, 1.0, seed=sd + 1))
984
+ if kind == "x_scale":
985
+ return (x, torch.tensor([0.125], device="cuda", dtype=dtype))
986
+ if kind == "ce":
987
+ g = torch.Generator(device="cuda").manual_seed(sd + 4)
988
+ return (x, torch.randint(0, N, (M,), generator=g, device="cuda", dtype=torch.int64))
989
+ if kind == "int8":
990
+ g = torch.Generator(device="cuda").manual_seed(sd)
991
+ q = torch.randint(-127, 127, (M, N), generator=g, device="cuda", dtype=torch.int8)
992
+ g2 = torch.Generator(device="cuda").manual_seed(sd + 1)
993
+ return (q, (torch.rand((M, 1), generator=g2, device="cuda") * 0.05 + 0.005).to(torch.float16))
994
+ # rope family: N is the head dim D (even)
995
+ D = N
996
+ if D % 2:
997
+ raise ValueError(f"{name}: head dim must be even, got {D}")
998
+ g = torch.Generator(device="cuda").manual_seed(sd + 5)
999
+ ang = torch.randn((M, D // 2), generator=g, device="cuda", dtype=torch.float32)
1000
+ c, s = torch.cos(ang), torch.sin(ang)
1001
+ if kind == "rope_inter":
1002
+ return (x, c.to(dtype), s.to(dtype))
1003
+ cos = torch.cat([c, c], -1).to(dtype)
1004
+ sin = torch.cat([s, s], -1).to(dtype)
1005
+ if kind == "rope":
1006
+ return (x, cos, sin)
1007
+ if kind == "qkr":
1008
+ return (x, _fixed((D,), dtype, 1.0, seed=sd + 1), cos, sin)
1009
+ if kind == "arr":
1010
+ return (x, _fixed((M, D), dtype, 1.0, seed=sd + 3), _fixed((D,), dtype, 1.0, seed=sd + 1), cos, sin)
1011
+ raise KeyError(kind)
1012
+
1013
+
1014
+ # ---- SHORT-ROW regime variants (V2.7 invention targets) -----------------------------------
1015
+ # The shape-grid characterized ONE loss region for the whole product: 16384x2048 (many short
1016
+ # rows), where row-per-program schedules underuse the GPU and inductor wins. These variants
1017
+ # are the SAME ops with the bench (and its correctness case) pinned INSIDE that region —
1018
+ # the invention question is whether RL finds a schedule family (split-row / multi-row /
1019
+ # persistent) that wins where its entire current style loses. Adversarial correctness sweep
1020
+ # unchanged (same make_inputs/stress).
1021
+ _SHORT_M, _SHORT_N = 16384, 2048
1022
+ _SHORT_BASES = ["rmsnorm", "softmax", "layernorm_gelu", "add_layernorm_sigmoid",
1023
+ # F1 falsification slate: the 10 worst remaining loss-cell ops — does the
1024
+ # whole-row style TRANSFER from rl_adapter_invent without new ideas?
1025
+ "add_layernorm_tanh", "add_layernorm_silu", "add_layernorm_gelu",
1026
+ "layernorm_sigmoid", "layernorm_tanh", "rmsnorm_silu", "add_rmsnorm_silu",
1027
+ "add_rmsnorm_sigmoid", "add_layernorm_square", "add_rmsnorm_gelu"]
1028
+
1029
+
1030
+ def _register_short_variants():
1031
+ for base in _SHORT_BASES:
1032
+ b = SPECS[base]
1033
+ name = base + "_short"
1034
+ if name in SPECS:
1035
+ continue
1036
+ SPECS[name] = OpSpec(
1037
+ name, b.reference, b.make_inputs,
1038
+ (lambda base=base: grid_inputs(base, _SHORT_M, _SHORT_N)),
1039
+ b.stress_inputs,
1040
+ b.signature_hint + "\n # REGIME: M=16384 rows of only N=2048. One-program-per-row "
1041
+ "underuses the GPU here;\n # consider processing MULTIPLE rows per program or "
1042
+ "splitting work differently.",
1043
+ notes=f"{base} pinned to the characterized loss regime (16384x2048) — schedule invention target.",
1044
+ extra=dict(b.extra))
1045
+
1046
+
1047
+ # ---- register the generative fusion-chain grammar (chains.py) ----------------------------
1048
+ def _register_chains():
1049
+ import chains
1050
+ _IN = {"rms": (_mk_rmsnorm, _bench_rmsnorm, _stress_rmsnorm),
1051
+ "add_rms": (_mk_add_rmsnorm, _bench_add_rmsnorm, _stress_add_rmsnorm),
1052
+ "ln": (_mk_layernorm, _bench_layernorm, _stress_layernorm),
1053
+ "add_ln": (_mk_add_layernorm, _bench_add_layernorm, _stress_add_layernorm)}
1054
+ _SIG = {"rms": "def run(x, w):", "add_rms": "def run(x, residual, w):",
1055
+ "ln": "def run(x, w, b):", "add_ln": "def run(x, residual, w, b):"}
1056
+ for name, kind, ref, _structs in chains.all_chains():
1057
+ if name in SPECS:
1058
+ continue
1059
+ mk, bench, stress = _IN[kind]
1060
+ act = name.rsplit("_", 1)[-1]
1061
+ SPECS[name] = OpSpec(name, ref, mk, bench, stress,
1062
+ f"{_SIG[kind]} # fused {name}: {kind.replace('add_','residual+').replace('rms','RMSNorm').replace('ln','LayerNorm')} "
1063
+ f"then {act} epilogue; accumulate the reduction in fp32",
1064
+ notes=f"generative fusion chain (reduction->epilogue): {name}.")
1065
+ _register_chains()
1066
+ _register_short_variants()