liangsu9988 commited on
Commit
36cbc8c
·
verified ·
1 Parent(s): ad4e599

Promote latest kernel artifacts to main

Browse files
README.md CHANGED
@@ -1,9 +1,151 @@
1
- # flashrt/adaptive-layernorm-producers
2
 
3
- This repository is a compatibility mirror for older `kernels` clients
4
- that resolve repositories through the default Hugging Face model repo API.
5
 
6
- Canonical Kernel Hub repo: https://huggingface.co/kernels/flashrt/adaptive-layernorm-producers
7
 
8
- Do not edit this mirror by hand. It is generated from the Kernel Hub
9
- `vN` branches and contains the same `build/**` artifacts.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # adaptive-layernorm-producers
2
 
3
+ FlashRT native CUDA producer kernels for DiT/Wan-style adaptive LayerNorm
4
+ blocks.
5
 
6
+ This package fuses the producer side of the block:
7
 
8
+ ```text
9
+ LayerNorm / adaptive LayerNorm modulation -> low-precision activation
10
+ ```
11
+
12
+ into a single Kernel Hub operator. It is intended for transformer and diffuser
13
+ runtimes where the following GEMM already consumes FP8 or NVFP4 activations.
14
+ Without this fusion, the runtime pays for separate normalization, modulation,
15
+ BF16 materialization, and quantization launches before the GEMM.
16
+
17
+ The source kernels are derived from the production FlashRT runtime:
18
+
19
+ - `official/FlashRT/csrc/quantize/ada_layer_norm_fp8.cu`
20
+ - `official/FlashRT/csrc/kernels/dit_bf16.cu::layer_norm_no_affine_fp8_static_bf16`
21
+
22
+ The full FlashRT model runtime and serving pipeline live upstream at
23
+ [LiangSu8899/FlashRT](https://github.com/LiangSu8899/FlashRT).
24
+
25
+ ## Available Functions
26
+
27
+ - `ada_layer_norm_quant_fp8_bf16(x, scale, shift, act_scale, eps=1e-5, out=None)`
28
+ - `ada_layer_norm_quant_fp8_ptok_bf16(x, scale, shift, act_scale, eps=1e-5, out=None)`
29
+ - `ada_layer_norm_quant_fp8_ptok_table_bf16(x, temb, table, act_scale, shift_idx, scale_idx, eps=1e-5, out=None)`
30
+ - `ada_layer_norm_quant_fp8_modfp8_bf16(x, scale_fp8, shift_fp8, scale_deq, shift_deq, act_scale, eps=1e-5, out=None)`
31
+ - `awq_ada_layer_norm_quant_fp8_bf16(x, scale, shift, inv_s, act_scale, eps=1e-5, out=None)`
32
+ - `ada_layer_norm_quant_nvfp4_swizzled_bf16(x, scale, shift, eps=1e-5, packed=None, sf_swizzled=None)`
33
+ - `ada_layer_norm_quant_nvfp4_swizzled_modfp8_bf16(x, scale_fp8, shift_fp8, scale_deq, shift_deq, eps=1e-5, packed=None, sf_swizzled=None)`
34
+ - `layer_norm_no_affine_quant_fp8_static_bf16(x, act_scale, eps=1e-5, out=None)`
35
+ - `layer_norm_no_affine_quant_nvfp4_swizzled_bf16(x, eps=1e-5, packed=None, sf_swizzled=None)`
36
+ - `adaln_modulation6_bf16(adaln_params, layer_modulation, out=None)`
37
+ - `swizzled_sf_size(rows, dim)`
38
+
39
+ ## Tensor Contract
40
+
41
+ - `x`: contiguous CUDA `torch.bfloat16`, shape `(rows, dim)`.
42
+ - Broadcast `scale`, `shift`, and `inv_s`: contiguous CUDA
43
+ `torch.bfloat16`, shape `(dim,)`.
44
+ - Per-token `scale` and `shift`: contiguous CUDA `torch.bfloat16`, shape
45
+ `(rows, dim)`.
46
+ - Table form `temb`: contiguous CUDA `torch.bfloat16`, shape
47
+ `(rows, chunks, dim)`; `table`: contiguous CUDA `torch.float32`, shape
48
+ `(chunks, dim)`.
49
+ - `scale_fp8`, `shift_fp8`: contiguous CUDA `torch.float8_e4m3fn`, shape `(dim,)`.
50
+ - `scale_deq`, `shift_deq`, `act_scale`: CUDA `torch.float32` scalar tensors.
51
+ - FP8 outputs use `torch.float8_e4m3fn`, shape `(rows, dim)`.
52
+ - NVFP4 packed output uses `torch.uint8`, shape `(rows, dim // 2)`.
53
+ - NVFP4 scale output uses the FlashRT/CUTLASS 128x4 swizzled uint8 scale
54
+ layout. Allocate with `swizzled_sf_size(rows, dim)`.
55
+ - `dim` must be even for FP8 outputs and divisible by 16 for NVFP4 outputs.
56
+ - The package targets CUDA 12.8+ and Blackwell-class deployment paths.
57
+ - `adaln_modulation6_bf16` takes FP32 `(batch, sequence, 6, dim)` parameters
58
+ plus FP32 `(6, dim)` layer modulation and returns six BF16
59
+ `(batch, sequence, dim)` tensors.
60
+
61
+ ## Minimal Usage
62
+
63
+ ```python
64
+ from kernels import get_kernel
65
+ import torch
66
+
67
+ ops = get_kernel("flashrt/adaptive-layernorm-producers", version=1, trust_remote_code=True)
68
+
69
+ rows, dim = 2520, 3072
70
+ x = torch.randn((rows, dim), device="cuda", dtype=torch.bfloat16)
71
+ scale = torch.zeros((dim,), device="cuda", dtype=torch.bfloat16)
72
+ shift = torch.zeros((dim,), device="cuda", dtype=torch.bfloat16)
73
+ act_scale = torch.tensor([0.025], device="cuda", dtype=torch.float32)
74
+
75
+ x_fp8 = ops.ada_layer_norm_quant_fp8_bf16(x, scale, shift, act_scale)
76
+ ```
77
+
78
+ Per-token modulation without materializing a broadcast buffer:
79
+
80
+ ```python
81
+ ptok_scale = torch.zeros_like(x)
82
+ ptok_shift = torch.zeros_like(x)
83
+ x_fp8 = ops.ada_layer_norm_quant_fp8_ptok_bf16(
84
+ x, ptok_scale, ptok_shift, act_scale
85
+ )
86
+ ```
87
+
88
+ Fused host modulation table add and chunk selection:
89
+
90
+ ```python
91
+ chunks = 6
92
+ temb = torch.zeros((rows, chunks, dim), device="cuda", dtype=torch.bfloat16)
93
+ table = torch.zeros((chunks, dim), device="cuda", dtype=torch.float32)
94
+ x_fp8 = ops.ada_layer_norm_quant_fp8_ptok_table_bf16(
95
+ x, temb, table, act_scale, shift_idx=0, scale_idx=1
96
+ )
97
+ ```
98
+
99
+ Static-buffer usage for CUDA Graph capture:
100
+
101
+ ```python
102
+ out = torch.empty_like(x, dtype=torch.float8_e4m3fn)
103
+ ops.ada_layer_norm_quant_fp8_bf16(x, scale, shift, act_scale, out=out)
104
+ ```
105
+
106
+ NVFP4 producer:
107
+
108
+ ```python
109
+ packed, sf = ops.ada_layer_norm_quant_nvfp4_swizzled_bf16(x, scale, shift)
110
+
111
+ # GROOT N1.7 pre-FFN producer on Thor.
112
+ packed_ln, sf_ln = ops.layer_norm_no_affine_quant_nvfp4_swizzled_bf16(x)
113
+ ```
114
+
115
+ Six-way DiT modulation:
116
+
117
+ ```python
118
+ params = torch.randn((1, rows, 6, dim), device="cuda", dtype=torch.float32)
119
+ modulation = torch.randn((6, dim), device="cuda", dtype=torch.float32)
120
+ outputs = ops.adaln_modulation6_bf16(params, modulation)
121
+ ```
122
+
123
+ ## Validation
124
+
125
+ ```bash
126
+ python adaptive-layernorm-producers/tests/test_adaptive_layernorm_producers.py --backend source --mode full
127
+ python adaptive-layernorm-producers/benchmarks/benchmark.py --backend source --iters 100
128
+ ```
129
+
130
+ The validation suite checks exact FP8 output for small producer shapes, exact
131
+ NVFP4 packed/scalefactor output against the native staged BF16-norm plus FP4
132
+ quantizer contract on SM110, and strict FP8
133
+ boundary accounting for long video shapes. The long-shape policy requires
134
+ `p99_abs == 0` and only a tiny count of adjacent FP8-code boundary differences
135
+ against the eager reference.
136
+
137
+ See `VALIDATION.md` and `benchmarks/RESULTS.md` for the current local RTX 5090
138
+ source-build results.
139
+
140
+ The SM110 FP4 producer uses the native GROOT N1.7 contract from FlashRT commit
141
+ `24df793f4fa2d50780aea03b644208c6e0cb4162`. Its scale conversion and FP4 bin
142
+ boundaries intentionally differ from the older SM120 producer; release tests
143
+ compare each architecture against its own production contract.
144
+
145
+ The aarch64 release variant `torch211-cxx11-cu130-aarch64-linux` contains
146
+ native SM87 and SM110a code objects. Its SM110a path was validated on NVIDIA
147
+ Thor from the exact Hub-downloaded artifact over decode, DiT, vision, and
148
+ 2K/4K video rows. The two per-token entry points were also compared directly
149
+ with their raw registered ops using A-B-B-A timing; all six wrapper/raw rows
150
+ were bit-exact and the measured ratio was `0.997-1.000`. SM87 runtime
151
+ validation remains pending on Jetson AGX Orin hardware.
benchmarks/RESULTS.md CHANGED
@@ -59,12 +59,3 @@ were bit-exact in every row.
59
 
60
  The table reports the uploaded SM87+SM110a fat binary on Thor. SM87 is present
61
  as native cubin code but is not included in these runtime measurements.
62
-
63
- ### GROOT N1.7 FP4 producer update
64
-
65
- The additive FP4 producer from FlashRT
66
- `24df793f4fa2d50780aea03b644208c6e0cb4162` was rebuilt as
67
- `torch213-cxx11-cu130-aarch64-linux` on the same Thor host. Packed FP4 values
68
- and SFA bytes were exact against the native staged contract. The Tensor
69
- wrapper/raw registered-op measurement was `8.3094/8.1932 us` (`1.0142x`) in
70
- direct mode and `5.3749/5.3797 us` (`0.9991x`) under CUDA Graph.
 
59
 
60
  The table reports the uploaded SM87+SM110a fat binary on Thor. SM87 is present
61
  as native cubin code but is not included in these runtime measurements.
 
 
 
 
 
 
 
 
 
benchmarks/benchmark.py CHANGED
@@ -14,6 +14,7 @@ import torch
14
  ROOT = Path(__file__).resolve().parents[2]
15
  sys.path.insert(0, str(ROOT / "adaptive-layernorm-producers" / "tests"))
16
  from test_adaptive_layernorm_producers import ( # noqa: E402
 
17
  load_source_ops,
18
  make_case,
19
  quant_fp8,
@@ -169,6 +170,45 @@ def run_case(ops, name: str, rows: int, dim: int, eps: float, iters: int) -> dic
169
  }
170
 
171
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
172
  def main() -> None:
173
  parser = argparse.ArgumentParser()
174
  parser.add_argument("--backend", choices=["source", "installed"], default="source")
@@ -190,6 +230,13 @@ def main() -> None:
190
  ("wan_video_4k", 4096, 3072),
191
  ]
192
  rows = [run_case(ops, name, r, d, 1e-5, args.iters) for name, r, d in shapes]
 
 
 
 
 
 
 
193
  modulation_rows = []
194
  for name, batch, sequence, dim in [
195
  ("groot_dit", 1, 51, 1536),
@@ -212,6 +259,19 @@ def main() -> None:
212
  )
213
  lines.append(line)
214
  print(line)
 
 
 
 
 
 
 
 
 
 
 
 
 
215
  lines.extend(
216
  [
217
  "",
 
14
  ROOT = Path(__file__).resolve().parents[2]
15
  sys.path.insert(0, str(ROOT / "adaptive-layernorm-producers" / "tests"))
16
  from test_adaptive_layernorm_producers import ( # noqa: E402
17
+ _nvfp4_out,
18
  load_source_ops,
19
  make_case,
20
  quant_fp8,
 
170
  }
171
 
172
 
173
+ def run_ptok_table(ops, name: str, rows: int, dim: int, eps: float, iters: int):
174
+ chunks = 6
175
+ x = torch.randn((rows, dim), device="cuda", dtype=torch.bfloat16)
176
+ temb = torch.randn(
177
+ (rows, chunks, dim), device="cuda", dtype=torch.bfloat16
178
+ )
179
+ table = torch.randn((chunks, dim), device="cuda", dtype=torch.float32)
180
+ out = torch.empty_like(x)
181
+ packed, sf = _nvfp4_out(x)
182
+
183
+ def fused_bf16():
184
+ return ops.ada_layer_norm_ptok_table_bf16(
185
+ x, temb, table, 0, 1, eps, out=out
186
+ )
187
+
188
+ def fused_nvfp4():
189
+ return ops.ada_layer_norm_quant_nvfp4_swizzled_ptok_table_bf16(
190
+ x, temb, table, 0, 1, eps, packed=packed, sf_swizzled=sf
191
+ )
192
+
193
+ def eager_bf16():
194
+ xf = x.float()
195
+ mean = xf.mean(dim=-1, keepdim=True)
196
+ var = (xf - mean).square().mean(dim=-1, keepdim=True)
197
+ norm = (xf - mean) * torch.rsqrt(var + eps)
198
+ shift = temb[:, 0].float() + table[0]
199
+ scale = temb[:, 1].float() + table[1]
200
+ return (norm * (1.0 + scale) + shift).to(torch.bfloat16)
201
+
202
+ return {
203
+ "shape": name,
204
+ "rows": rows,
205
+ "dim": dim,
206
+ "bf16_us": time_cuda(fused_bf16, iters=iters),
207
+ "eager_us": time_cuda(eager_bf16, iters=iters),
208
+ "nvfp4_us": time_cuda(fused_nvfp4, iters=iters),
209
+ }
210
+
211
+
212
  def main() -> None:
213
  parser = argparse.ArgumentParser()
214
  parser.add_argument("--backend", choices=["source", "installed"], default="source")
 
230
  ("wan_video_4k", 4096, 3072),
231
  ]
232
  rows = [run_case(ops, name, r, d, 1e-5, args.iters) for name, r, d in shapes]
233
+ table_rows = [
234
+ run_ptok_table(ops, name, r, d, 1e-5, args.iters)
235
+ for name, r, d in [
236
+ ("video_short", 64, 3072),
237
+ ("video_long", 2520, 3072),
238
+ ]
239
+ ]
240
  modulation_rows = []
241
  for name, batch, sequence, dim in [
242
  ("groot_dit", 1, 51, 1536),
 
259
  )
260
  lines.append(line)
261
  print(line)
262
+ lines.extend([
263
+ "",
264
+ "| Per-token table shape | BF16 fused us | BF16 eager us | Speedup | NVFP4 fused us |",
265
+ "|---|---:|---:|---:|---:|",
266
+ ])
267
+ for row in table_rows:
268
+ line = (
269
+ f"| {row['shape']} M{row['rows']} D{row['dim']} | "
270
+ f"{row['bf16_us']:.3f} | {row['eager_us']:.3f} | "
271
+ f"{row['eager_us'] / row['bf16_us']:.2f}x | {row['nvfp4_us']:.3f} |"
272
+ )
273
+ lines.append(line)
274
+ print(line)
275
  lines.extend(
276
  [
277
  "",
build/torch211-cxx11-cu128-x86_64-linux/__init__.py CHANGED
@@ -105,6 +105,26 @@ def _ada_layer_norm_quant_fp8_ptok_table_bf16_fake(
105
  return None
106
 
107
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
108
  @torch.library.register_fake(add_op_namespace_prefix("ada_layer_norm_quant_fp8_modfp8_bf16"))
109
  def _ada_layer_norm_quant_fp8_modfp8_bf16_fake(
110
  x: torch.Tensor,
@@ -162,6 +182,32 @@ def _ada_layer_norm_quant_nvfp4_swizzled_bf16_fake(
162
  return None
163
 
164
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
165
  @torch.library.register_fake(add_op_namespace_prefix("ada_layer_norm_quant_nvfp4_swizzled_modfp8_bf16"))
166
  def _ada_layer_norm_quant_nvfp4_swizzled_modfp8_bf16_fake(
167
  x: torch.Tensor,
@@ -198,6 +244,22 @@ def _layer_norm_no_affine_quant_fp8_static_bf16_fake(
198
  return None
199
 
200
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
201
  def ada_layer_norm_quant_fp8_bf16(
202
  x: torch.Tensor,
203
  scale: torch.Tensor,
@@ -266,6 +328,25 @@ def ada_layer_norm_quant_fp8_ptok_table_bf16(
266
  return out
267
 
268
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
269
  def ada_layer_norm_quant_fp8_modfp8_bf16(
270
  x: torch.Tensor,
271
  scale_fp8: torch.Tensor,
@@ -319,6 +400,27 @@ def ada_layer_norm_quant_nvfp4_swizzled_bf16(
319
  return packed, sf_swizzled
320
 
321
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
322
  def ada_layer_norm_quant_nvfp4_swizzled_modfp8_bf16(
323
  x: torch.Tensor,
324
  scale_fp8: torch.Tensor,
@@ -353,6 +455,21 @@ def layer_norm_no_affine_quant_fp8_static_bf16(
353
  return out
354
 
355
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
356
  @torch.library.register_fake(add_op_namespace_prefix("adaln_modulation6_bf16"))
357
  def _adaln_modulation6_bf16_fake(
358
  adaln_params: torch.Tensor,
@@ -413,11 +530,14 @@ __all__ = [
413
  "ada_layer_norm_quant_fp8_bf16",
414
  "ada_layer_norm_quant_fp8_ptok_bf16",
415
  "ada_layer_norm_quant_fp8_ptok_table_bf16",
 
416
  "ada_layer_norm_quant_fp8_modfp8_bf16",
417
  "awq_ada_layer_norm_quant_fp8_bf16",
418
  "ada_layer_norm_quant_nvfp4_swizzled_bf16",
 
419
  "ada_layer_norm_quant_nvfp4_swizzled_modfp8_bf16",
420
  "layer_norm_no_affine_quant_fp8_static_bf16",
 
421
  "adaln_modulation6_bf16",
422
  "swizzled_sf_size",
423
  ]
 
105
  return None
106
 
107
 
108
+ @torch.library.register_fake(add_op_namespace_prefix("ada_layer_norm_ptok_table_bf16"))
109
+ def _ada_layer_norm_ptok_table_bf16_fake(
110
+ x: torch.Tensor,
111
+ temb: torch.Tensor,
112
+ table: torch.Tensor,
113
+ shift_idx: int,
114
+ scale_idx: int,
115
+ eps: float,
116
+ out: torch.Tensor,
117
+ ) -> None:
118
+ _check_x(x)
119
+ if temb.dim() != 3 or temb.shape[0] != x.shape[0] or temb.shape[2] != x.shape[1]:
120
+ raise RuntimeError("temb must have shape (seq_len, n_chunks, dim)")
121
+ if table.shape != (temb.shape[1], x.shape[1]):
122
+ raise RuntimeError("table must have shape (n_chunks, dim)")
123
+ if out.shape != x.shape:
124
+ raise RuntimeError("out must have the same shape as x")
125
+ return None
126
+
127
+
128
  @torch.library.register_fake(add_op_namespace_prefix("ada_layer_norm_quant_fp8_modfp8_bf16"))
129
  def _ada_layer_norm_quant_fp8_modfp8_bf16_fake(
130
  x: torch.Tensor,
 
182
  return None
183
 
184
 
185
+ @torch.library.register_fake(
186
+ add_op_namespace_prefix("ada_layer_norm_quant_nvfp4_swizzled_ptok_table_bf16")
187
+ )
188
+ def _ada_layer_norm_quant_nvfp4_swizzled_ptok_table_bf16_fake(
189
+ x: torch.Tensor,
190
+ temb: torch.Tensor,
191
+ table: torch.Tensor,
192
+ shift_idx: int,
193
+ scale_idx: int,
194
+ eps: float,
195
+ packed: torch.Tensor,
196
+ sf_swizzled: torch.Tensor,
197
+ ) -> None:
198
+ _check_x(x)
199
+ if temb.dim() != 3 or temb.shape[0] != x.shape[0] or temb.shape[2] != x.shape[1]:
200
+ raise RuntimeError("temb must have shape (seq_len, n_chunks, dim)")
201
+ if table.shape != (temb.shape[1], x.shape[1]):
202
+ raise RuntimeError("table must have shape (n_chunks, dim)")
203
+ rows, dim = x.shape
204
+ if packed.shape != (rows, dim // 2):
205
+ raise RuntimeError("packed must have shape (rows, dim // 2)")
206
+ if sf_swizzled.numel() < swizzled_sf_size(rows, dim):
207
+ raise RuntimeError("sf_swizzled is too small")
208
+ return None
209
+
210
+
211
  @torch.library.register_fake(add_op_namespace_prefix("ada_layer_norm_quant_nvfp4_swizzled_modfp8_bf16"))
212
  def _ada_layer_norm_quant_nvfp4_swizzled_modfp8_bf16_fake(
213
  x: torch.Tensor,
 
244
  return None
245
 
246
 
247
+ @torch.library.register_fake(add_op_namespace_prefix("layer_norm_no_affine_quant_nvfp4_swizzled_bf16"))
248
+ def _layer_norm_no_affine_quant_nvfp4_swizzled_bf16_fake(
249
+ x: torch.Tensor,
250
+ eps: float,
251
+ packed: torch.Tensor,
252
+ sf_swizzled: torch.Tensor,
253
+ ) -> None:
254
+ _check_x(x)
255
+ rows, dim = x.shape
256
+ if packed.shape != (rows, dim // 2):
257
+ raise RuntimeError("packed must have shape (rows, dim // 2)")
258
+ if sf_swizzled.numel() < swizzled_sf_size(rows, dim):
259
+ raise RuntimeError("sf_swizzled is too small")
260
+ return None
261
+
262
+
263
  def ada_layer_norm_quant_fp8_bf16(
264
  x: torch.Tensor,
265
  scale: torch.Tensor,
 
328
  return out
329
 
330
 
331
+ def ada_layer_norm_ptok_table_bf16(
332
+ x: torch.Tensor,
333
+ temb: torch.Tensor,
334
+ table: torch.Tensor,
335
+ shift_idx: int,
336
+ scale_idx: int,
337
+ eps: float = 1e-5,
338
+ out: torch.Tensor | None = None,
339
+ ) -> torch.Tensor:
340
+ """Fuse table add, chunk selection and AdaLN into a BF16 output."""
341
+
342
+ if out is None:
343
+ out = torch.empty_like(x)
344
+ ops.ada_layer_norm_ptok_table_bf16(
345
+ x, temb, table, int(shift_idx), int(scale_idx), float(eps), out
346
+ )
347
+ return out
348
+
349
+
350
  def ada_layer_norm_quant_fp8_modfp8_bf16(
351
  x: torch.Tensor,
352
  scale_fp8: torch.Tensor,
 
400
  return packed, sf_swizzled
401
 
402
 
403
+ def ada_layer_norm_quant_nvfp4_swizzled_ptok_table_bf16(
404
+ x: torch.Tensor,
405
+ temb: torch.Tensor,
406
+ table: torch.Tensor,
407
+ shift_idx: int,
408
+ scale_idx: int,
409
+ eps: float = 1e-5,
410
+ packed: torch.Tensor | None = None,
411
+ sf_swizzled: torch.Tensor | None = None,
412
+ ) -> tuple[torch.Tensor, torch.Tensor]:
413
+ """Fuse per-token table AdaLN into NVFP4 and CUTLASS scale factors."""
414
+
415
+ if packed is None or sf_swizzled is None:
416
+ packed, sf_swizzled = _alloc_nvfp4(x)
417
+ ops.ada_layer_norm_quant_nvfp4_swizzled_ptok_table_bf16(
418
+ x, temb, table, int(shift_idx), int(scale_idx), float(eps),
419
+ packed, sf_swizzled
420
+ )
421
+ return packed, sf_swizzled
422
+
423
+
424
  def ada_layer_norm_quant_nvfp4_swizzled_modfp8_bf16(
425
  x: torch.Tensor,
426
  scale_fp8: torch.Tensor,
 
455
  return out
456
 
457
 
458
+ def layer_norm_no_affine_quant_nvfp4_swizzled_bf16(
459
+ x: torch.Tensor,
460
+ eps: float = 1e-5,
461
+ packed: torch.Tensor | None = None,
462
+ sf_swizzled: torch.Tensor | None = None,
463
+ ) -> tuple[torch.Tensor, torch.Tensor]:
464
+ """SM110 no-affine LayerNorm producing NVFP4 and CUTLASS SFA."""
465
+ if packed is None or sf_swizzled is None:
466
+ packed, sf_swizzled = _alloc_nvfp4(x)
467
+ ops.layer_norm_no_affine_quant_nvfp4_swizzled_bf16(
468
+ x, float(eps), packed, sf_swizzled
469
+ )
470
+ return packed, sf_swizzled
471
+
472
+
473
  @torch.library.register_fake(add_op_namespace_prefix("adaln_modulation6_bf16"))
474
  def _adaln_modulation6_bf16_fake(
475
  adaln_params: torch.Tensor,
 
530
  "ada_layer_norm_quant_fp8_bf16",
531
  "ada_layer_norm_quant_fp8_ptok_bf16",
532
  "ada_layer_norm_quant_fp8_ptok_table_bf16",
533
+ "ada_layer_norm_ptok_table_bf16",
534
  "ada_layer_norm_quant_fp8_modfp8_bf16",
535
  "awq_ada_layer_norm_quant_fp8_bf16",
536
  "ada_layer_norm_quant_nvfp4_swizzled_bf16",
537
+ "ada_layer_norm_quant_nvfp4_swizzled_ptok_table_bf16",
538
  "ada_layer_norm_quant_nvfp4_swizzled_modfp8_bf16",
539
  "layer_norm_no_affine_quant_fp8_static_bf16",
540
+ "layer_norm_no_affine_quant_nvfp4_swizzled_bf16",
541
  "adaln_modulation6_bf16",
542
  "swizzled_sf_size",
543
  ]
build/torch211-cxx11-cu128-x86_64-linux/{_adaptive_layernorm_producers_cuda_9b532b4.abi3.so → _adaptive_layernorm_producers_cuda_bb9b4e5.abi3.so} RENAMED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:1fe736aa20ba67f33f19e7e4ec02d3eebcdf4427f6240dfd7d57be30f37fbf6a
3
- size 1891648
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:646d65e6da951010d153d47c3dbe8e78881b0f670f115cd6e1f63bbea239cbfd
3
+ size 2000416
build/torch211-cxx11-cu128-x86_64-linux/_ops.py CHANGED
@@ -1,9 +1,9 @@
1
  import torch
2
- from . import _adaptive_layernorm_producers_cuda_9b532b4
3
- ops = torch.ops._adaptive_layernorm_producers_cuda_9b532b4
4
 
5
  def add_op_namespace_prefix(op_name: str):
6
  """
7
  Prefix op by namespace.
8
  """
9
- return f"_adaptive_layernorm_producers_cuda_9b532b4::{op_name}"
 
1
  import torch
2
+ from . import _adaptive_layernorm_producers_cuda_bb9b4e5
3
+ ops = torch.ops._adaptive_layernorm_producers_cuda_bb9b4e5
4
 
5
  def add_op_namespace_prefix(op_name: str):
6
  """
7
  Prefix op by namespace.
8
  """
9
+ return f"_adaptive_layernorm_producers_cuda_bb9b4e5::{op_name}"
build/torch211-cxx11-cu128-x86_64-linux/metadata.json CHANGED
@@ -1,6 +1,6 @@
1
  {
2
  "name": "adaptive-layernorm-producers",
3
- "id": "_adaptive_layernorm_producers_cuda_9b532b4",
4
  "version": 1,
5
  "license": "Apache-2.0",
6
  "python-depends": [],
@@ -19,5 +19,25 @@
19
  "8.9",
20
  "9.0"
21
  ]
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
22
  }
23
- }
 
1
  {
2
  "name": "adaptive-layernorm-producers",
3
+ "id": "_adaptive_layernorm_producers_cuda_bb9b4e5",
4
  "version": 1,
5
  "license": "Apache-2.0",
6
  "python-depends": [],
 
19
  "8.9",
20
  "9.0"
21
  ]
22
+ },
23
+ "digest": {
24
+ "algorithm": "sha256",
25
+ "files": {
26
+ "__init__.py": "+ZwPYHOKPDe9jTo0jlUuIRwuU4XT1gxfK/Bxk+ZLkNI=",
27
+ "_adaptive_layernorm_producers_cuda_bb9b4e5.abi3.so": "ZG1l5tqVEBDRU9R8Pb6OeIgbD2cPEVzW4fY7vqI5y/0=",
28
+ "_ops.py": "M1g8M15LM5VNF7VWdjaC0SGIFEv/Lu08K3trrsZPuSE=",
29
+ "adaptive_layernorm_producers/__init__.py": "DFYPlrhXwYjEqCl/8n0SmWGZV8NFml5DPhMjKfv98GY="
30
+ }
31
+ },
32
+ "provenance": {
33
+ "kernel-builder": {
34
+ "version": "0.17.0-dev0",
35
+ "sha": "b39ca23f1b36383df00b27b3ffe1276cd5dbea85",
36
+ "dirty": false
37
+ },
38
+ "kernel": {
39
+ "sha": "bb9b4e57f62b1e00f4c87262303558a6d8d5c61a",
40
+ "dirty": false
41
+ }
42
  }
43
+ }