liangsu9988 commited on
Commit
ad4e599
·
verified ·
1 Parent(s): 5b0d2e6

Initialize legacy kernels compatibility mirror

Browse files
Files changed (1) hide show
  1. README.md +6 -148
README.md CHANGED
@@ -1,151 +1,9 @@
1
- # adaptive-layernorm-producers
2
 
3
- FlashRT native CUDA producer kernels for DiT/Wan-style adaptive LayerNorm
4
- blocks.
5
 
6
- This package fuses the producer side of the block:
7
 
8
- ```text
9
- LayerNorm / adaptive LayerNorm modulation -> low-precision activation
10
- ```
11
-
12
- into a single Kernel Hub operator. It is intended for transformer and diffuser
13
- runtimes where the following GEMM already consumes FP8 or NVFP4 activations.
14
- Without this fusion, the runtime pays for separate normalization, modulation,
15
- BF16 materialization, and quantization launches before the GEMM.
16
-
17
- The source kernels are derived from the production FlashRT runtime:
18
-
19
- - `official/FlashRT/csrc/quantize/ada_layer_norm_fp8.cu`
20
- - `official/FlashRT/csrc/kernels/dit_bf16.cu::layer_norm_no_affine_fp8_static_bf16`
21
-
22
- The full FlashRT model runtime and serving pipeline live upstream at
23
- [LiangSu8899/FlashRT](https://github.com/LiangSu8899/FlashRT).
24
-
25
- ## Available Functions
26
-
27
- - `ada_layer_norm_quant_fp8_bf16(x, scale, shift, act_scale, eps=1e-5, out=None)`
28
- - `ada_layer_norm_quant_fp8_ptok_bf16(x, scale, shift, act_scale, eps=1e-5, out=None)`
29
- - `ada_layer_norm_quant_fp8_ptok_table_bf16(x, temb, table, act_scale, shift_idx, scale_idx, eps=1e-5, out=None)`
30
- - `ada_layer_norm_quant_fp8_modfp8_bf16(x, scale_fp8, shift_fp8, scale_deq, shift_deq, act_scale, eps=1e-5, out=None)`
31
- - `awq_ada_layer_norm_quant_fp8_bf16(x, scale, shift, inv_s, act_scale, eps=1e-5, out=None)`
32
- - `ada_layer_norm_quant_nvfp4_swizzled_bf16(x, scale, shift, eps=1e-5, packed=None, sf_swizzled=None)`
33
- - `ada_layer_norm_quant_nvfp4_swizzled_modfp8_bf16(x, scale_fp8, shift_fp8, scale_deq, shift_deq, eps=1e-5, packed=None, sf_swizzled=None)`
34
- - `layer_norm_no_affine_quant_fp8_static_bf16(x, act_scale, eps=1e-5, out=None)`
35
- - `layer_norm_no_affine_quant_nvfp4_swizzled_bf16(x, eps=1e-5, packed=None, sf_swizzled=None)`
36
- - `adaln_modulation6_bf16(adaln_params, layer_modulation, out=None)`
37
- - `swizzled_sf_size(rows, dim)`
38
-
39
- ## Tensor Contract
40
-
41
- - `x`: contiguous CUDA `torch.bfloat16`, shape `(rows, dim)`.
42
- - Broadcast `scale`, `shift`, and `inv_s`: contiguous CUDA
43
- `torch.bfloat16`, shape `(dim,)`.
44
- - Per-token `scale` and `shift`: contiguous CUDA `torch.bfloat16`, shape
45
- `(rows, dim)`.
46
- - Table form `temb`: contiguous CUDA `torch.bfloat16`, shape
47
- `(rows, chunks, dim)`; `table`: contiguous CUDA `torch.float32`, shape
48
- `(chunks, dim)`.
49
- - `scale_fp8`, `shift_fp8`: contiguous CUDA `torch.float8_e4m3fn`, shape `(dim,)`.
50
- - `scale_deq`, `shift_deq`, `act_scale`: CUDA `torch.float32` scalar tensors.
51
- - FP8 outputs use `torch.float8_e4m3fn`, shape `(rows, dim)`.
52
- - NVFP4 packed output uses `torch.uint8`, shape `(rows, dim // 2)`.
53
- - NVFP4 scale output uses the FlashRT/CUTLASS 128x4 swizzled uint8 scale
54
- layout. Allocate with `swizzled_sf_size(rows, dim)`.
55
- - `dim` must be even for FP8 outputs and divisible by 16 for NVFP4 outputs.
56
- - The package targets CUDA 12.8+ and Blackwell-class deployment paths.
57
- - `adaln_modulation6_bf16` takes FP32 `(batch, sequence, 6, dim)` parameters
58
- plus FP32 `(6, dim)` layer modulation and returns six BF16
59
- `(batch, sequence, dim)` tensors.
60
-
61
- ## Minimal Usage
62
-
63
- ```python
64
- from kernels import get_kernel
65
- import torch
66
-
67
- ops = get_kernel("flashrt/adaptive-layernorm-producers", version=1, trust_remote_code=True)
68
-
69
- rows, dim = 2520, 3072
70
- x = torch.randn((rows, dim), device="cuda", dtype=torch.bfloat16)
71
- scale = torch.zeros((dim,), device="cuda", dtype=torch.bfloat16)
72
- shift = torch.zeros((dim,), device="cuda", dtype=torch.bfloat16)
73
- act_scale = torch.tensor([0.025], device="cuda", dtype=torch.float32)
74
-
75
- x_fp8 = ops.ada_layer_norm_quant_fp8_bf16(x, scale, shift, act_scale)
76
- ```
77
-
78
- Per-token modulation without materializing a broadcast buffer:
79
-
80
- ```python
81
- ptok_scale = torch.zeros_like(x)
82
- ptok_shift = torch.zeros_like(x)
83
- x_fp8 = ops.ada_layer_norm_quant_fp8_ptok_bf16(
84
- x, ptok_scale, ptok_shift, act_scale
85
- )
86
- ```
87
-
88
- Fused host modulation table add and chunk selection:
89
-
90
- ```python
91
- chunks = 6
92
- temb = torch.zeros((rows, chunks, dim), device="cuda", dtype=torch.bfloat16)
93
- table = torch.zeros((chunks, dim), device="cuda", dtype=torch.float32)
94
- x_fp8 = ops.ada_layer_norm_quant_fp8_ptok_table_bf16(
95
- x, temb, table, act_scale, shift_idx=0, scale_idx=1
96
- )
97
- ```
98
-
99
- Static-buffer usage for CUDA Graph capture:
100
-
101
- ```python
102
- out = torch.empty_like(x, dtype=torch.float8_e4m3fn)
103
- ops.ada_layer_norm_quant_fp8_bf16(x, scale, shift, act_scale, out=out)
104
- ```
105
-
106
- NVFP4 producer:
107
-
108
- ```python
109
- packed, sf = ops.ada_layer_norm_quant_nvfp4_swizzled_bf16(x, scale, shift)
110
-
111
- # GROOT N1.7 pre-FFN producer on Thor.
112
- packed_ln, sf_ln = ops.layer_norm_no_affine_quant_nvfp4_swizzled_bf16(x)
113
- ```
114
-
115
- Six-way DiT modulation:
116
-
117
- ```python
118
- params = torch.randn((1, rows, 6, dim), device="cuda", dtype=torch.float32)
119
- modulation = torch.randn((6, dim), device="cuda", dtype=torch.float32)
120
- outputs = ops.adaln_modulation6_bf16(params, modulation)
121
- ```
122
-
123
- ## Validation
124
-
125
- ```bash
126
- python adaptive-layernorm-producers/tests/test_adaptive_layernorm_producers.py --backend source --mode full
127
- python adaptive-layernorm-producers/benchmarks/benchmark.py --backend source --iters 100
128
- ```
129
-
130
- The validation suite checks exact FP8 output for small producer shapes, exact
131
- NVFP4 packed/scalefactor output against the native staged BF16-norm plus FP4
132
- quantizer contract on SM110, and strict FP8
133
- boundary accounting for long video shapes. The long-shape policy requires
134
- `p99_abs == 0` and only a tiny count of adjacent FP8-code boundary differences
135
- against the eager reference.
136
-
137
- See `VALIDATION.md` and `benchmarks/RESULTS.md` for the current local RTX 5090
138
- source-build results.
139
-
140
- The SM110 FP4 producer uses the native GROOT N1.7 contract from FlashRT commit
141
- `24df793f4fa2d50780aea03b644208c6e0cb4162`. Its scale conversion and FP4 bin
142
- boundaries intentionally differ from the older SM120 producer; release tests
143
- compare each architecture against its own production contract.
144
-
145
- The aarch64 release variant `torch211-cxx11-cu130-aarch64-linux` contains
146
- native SM87 and SM110a code objects. Its SM110a path was validated on NVIDIA
147
- Thor from the exact Hub-downloaded artifact over decode, DiT, vision, and
148
- 2K/4K video rows. The two per-token entry points were also compared directly
149
- with their raw registered ops using A-B-B-A timing; all six wrapper/raw rows
150
- were bit-exact and the measured ratio was `0.997-1.000`. SM87 runtime
151
- validation remains pending on Jetson AGX Orin hardware.
 
1
+ # flashrt/adaptive-layernorm-producers
2
 
3
+ This repository is a compatibility mirror for older `kernels` clients
4
+ that resolve repositories through the default Hugging Face model repo API.
5
 
6
+ Canonical Kernel Hub repo: https://huggingface.co/kernels/flashrt/adaptive-layernorm-producers
7
 
8
+ Do not edit this mirror by hand. It is generated from the Kernel Hub
9
+ `vN` branches and contains the same `build/**` artifacts.