Hugging Face's logo

Kernels:
phanerozoic
/
quant-matmul

Kernel card Files
xet
 Community
phanerozoic commited on
Commit
e4a7eb7
·
verified ·
1 Parent(s): 1d5b00b

Add card

Browse files
Files changed (2) hide show
  1. CARD.md +108 -0
  2. README.md +108 -0
CARD.md ADDED
@@ -0,0 +1,108 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: kernels
3
+ license: apache-2.0
4
+ ---
5
+
6
+ # quant-matmul
7
+
8
+ W4A8 / W8A8 / FP16-storage GEMV-GEMM for quantized LLM inference on CPUs,
9
+ aarch64-first. Int4 weights in GPTQ/AWQ-compatible group quantization run
10
+ against per-token INT8 activations. Companions on the Hub:
11
+ [phanerozoic/bitnet-cpu](https://huggingface.co/kernels/phanerozoic/bitnet-cpu)
12
+ (ternary linears),
13
+ [phanerozoic/cpu-attn](https://huggingface.co/kernels/phanerozoic/cpu-attn)
14
+ (int8 KV attention),
15
+ [phanerozoic/decode-ops](https://huggingface.co/kernels/phanerozoic/decode-ops)
16
+ (fused glue and sampling).
17
+
18
+ Per group `g` of `group_size` weights along K:
19
+
20
+ y[m, n] = sa[m] * sum_g s[n, g] * (dot_g(a_m, w_n) - z[n, g] * asum_g[m])
21
+
22
+ Group dots are exact in int32 (nibble <= 15, |a| <= 127); groups accumulate
23
+ in fp32 in fixed order, so every dispatch tier produces bit-identical f32
24
+ output. Nibbles are shifted and masked in registers against a one-time
25
+ even/odd split of the activations; no unpack buffer exists.
26
+
27
+ | path | instruction | selected when |
28
+ |---|---|---|
29
+ | SDOT | `sdot` (dotprod) | aarch64 with `asimddp`: Cortex-A76+ (Pi 5), Neoverse, Apple silicon |
30
+ | NEON | `smull` + `sadalp` | any other aarch64 (Pi 4, Pi Zero 2) |
31
+ | scalar | portable C++ | x86-64 and everything else |
32
+
33
+ Chosen once at runtime from HWCAP; `QM_CPU_ISA` (`scalar`, `neon`,
34
+ `neondot`) demotes the selection for A/B runs; it never promotes.
35
+
36
+ ## Usage
37
+
38
+ ```python
39
+ import torch
40
+ from kernels import get_kernel
41
+
42
+ qm = get_kernel("phanerozoic/quant-matmul", version=1, trust_remote_code=True)
43
+
44
+ # From a GPTQ checkpoint's tensors:
45
+ packed, s, z, gs, perm = qm.from_gptq(qweight, qzeros, scales, g_idx)
46
+ y = qm.w4a8_linear(x, packed, s, z, gs, perm) # [..., N]
47
+
48
+ # Or as a module:
49
+ layer = qm.QuantLinear.from_gptq_tensors(qweight, qzeros, scales, g_idx)
50
+ y = layer(x)
51
+ ```
52
+
53
+ `version` selects the release branch; `trust_remote_code` is required by
54
+ `kernels` for publishers without the trusted-publisher mark.
55
+
56
+ ## API
57
+
58
+ | Function | Purpose |
59
+ |---|---|
60
+ | `pack_w4(intw, scales, zeros, group_size)` | raw nibbles `[N, K]` -> native packing |
61
+ | `from_gptq(qweight, qzeros, scales, g_idx=None, zero_offset=1)` | GPTQ tensors -> native; act-order returns an activation `perm` |
62
+ | `from_awq(qweight, qzeros, scales)` | AutoAWQ GEMM tensors -> native |
63
+ | `dequant_w4(packed, scales, zeros, group_size)` | reference f32 dequantization |
64
+ | `quantize_activation(x)` | per-token absmax INT8 |
65
+ | `w4a8_linear(x, packed, scales, zeros, group_size, perm)` | int4 weights x int8 activations |
66
+ | `w8a8_linear(x, w_int8, scale_wt)` | int8 x int8, per-row weight scale |
67
+ | `fp16_gemv(x, w_f16)` | f16-storage weights widened to f32 in registers |
68
+ | `QuantLinear` | `nn.Module` wrapper |
69
+
70
+ `zero_offset=1` follows the AutoGPTQ storage convention (the stored zero
71
+ plus one is subtracted at dequant); pass 0 for checkpoints without it.
72
+
73
+ ## Performance
74
+
75
+ Raspberry Pi 5 (4x Cortex-A76 2.4 GHz, SDOT path) and Raspberry Pi 4
76
+ Model B (4x Cortex-A72 1.8 GHz, NEON path), 64-bit Raspberry Pi OS,
77
+ torch 2.13 CPU, group_size 128:
78
+
79
+ | op, shape (M, N, K) | Pi 5 | Pi 4 |
80
+ |---|---|---|
81
+ | w4a8 1, 4096, 4096 | 1.16 ms | 4.08 ms |
82
+ | w4a8 1, 11008, 4096 | 2.98 ms | 11.7 ms |
83
+ | w4a8 1, 4096, 11008 | 3.57 ms | 10.4 ms |
84
+ | w4a8 128, 4096, 4096 | 23.2 ms | 209 ms |
85
+ | w8a8 1, 4096, 4096 | 1.46 ms | 5.68 ms |
86
+ | fp16 1, 4096, 4096 | 2.80 ms | 13.5 ms |
87
+
88
+ Decode (M=1) streams packed weights at 8-12 GB/s on the Pi 5, the board's
89
+ memory ceiling; M=128 runs 185 GOPS on the SDOT tier.
90
+
91
+ ## Requirements
92
+
93
+ - K divisible by group_size; group_size a multiple of 32 (64 and 128 tested).
94
+ - bf16 or f32 activations; output follows the input dtype.
95
+ - GPTQ act-order (`g_idx`) is supported through an activation permutation
96
+ applied per forward.
97
+ - x86-64 runs the scalar path: correct, not tuned.
98
+
99
+ ## Scope
100
+
101
+ Hub-loadable quantized linears for transformers-on-CPU, aimed at boards
102
+ where a dedicated llama.cpp runtime was previously the only usable path.
103
+ llama.cpp remains faster as a standalone runtime; this pack is for models
104
+ composed in Python.
105
+
106
+ ## License
107
+
108
+ Apache-2.0.
README.md ADDED
@@ -0,0 +1,108 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: kernels
3
+ license: apache-2.0
4
+ ---
5
+
6
+ # quant-matmul
7
+
8
+ W4A8 / W8A8 / FP16-storage GEMV-GEMM for quantized LLM inference on CPUs,
9
+ aarch64-first. Int4 weights in GPTQ/AWQ-compatible group quantization run
10
+ against per-token INT8 activations. Companions on the Hub:
11
+ [phanerozoic/bitnet-cpu](https://huggingface.co/kernels/phanerozoic/bitnet-cpu)
12
+ (ternary linears),
13
+ [phanerozoic/cpu-attn](https://huggingface.co/kernels/phanerozoic/cpu-attn)
14
+ (int8 KV attention),
15
+ [phanerozoic/decode-ops](https://huggingface.co/kernels/phanerozoic/decode-ops)
16
+ (fused glue and sampling).
17
+
18
+ Per group `g` of `group_size` weights along K:
19
+
20
+ y[m, n] = sa[m] * sum_g s[n, g] * (dot_g(a_m, w_n) - z[n, g] * asum_g[m])
21
+
22
+ Group dots are exact in int32 (nibble <= 15, |a| <= 127); groups accumulate
23
+ in fp32 in fixed order, so every dispatch tier produces bit-identical f32
24
+ output. Nibbles are shifted and masked in registers against a one-time
25
+ even/odd split of the activations; no unpack buffer exists.
26
+
27
+ | path | instruction | selected when |
28
+ |---|---|---|
29
+ | SDOT | `sdot` (dotprod) | aarch64 with `asimddp`: Cortex-A76+ (Pi 5), Neoverse, Apple silicon |
30
+ | NEON | `smull` + `sadalp` | any other aarch64 (Pi 4, Pi Zero 2) |
31
+ | scalar | portable C++ | x86-64 and everything else |
32
+
33
+ Chosen once at runtime from HWCAP; `QM_CPU_ISA` (`scalar`, `neon`,
34
+ `neondot`) demotes the selection for A/B runs; it never promotes.
35
+
36
+ ## Usage
37
+
38
+ ```python
39
+ import torch
40
+ from kernels import get_kernel
41
+
42
+ qm = get_kernel("phanerozoic/quant-matmul", version=1, trust_remote_code=True)
43
+
44
+ # From a GPTQ checkpoint's tensors:
45
+ packed, s, z, gs, perm = qm.from_gptq(qweight, qzeros, scales, g_idx)
46
+ y = qm.w4a8_linear(x, packed, s, z, gs, perm) # [..., N]
47
+
48
+ # Or as a module:
49
+ layer = qm.QuantLinear.from_gptq_tensors(qweight, qzeros, scales, g_idx)
50
+ y = layer(x)
51
+ ```
52
+
53
+ `version` selects the release branch; `trust_remote_code` is required by
54
+ `kernels` for publishers without the trusted-publisher mark.
55
+
56
+ ## API
57
+
58
+ | Function | Purpose |
59
+ |---|---|
60
+ | `pack_w4(intw, scales, zeros, group_size)` | raw nibbles `[N, K]` -> native packing |
61
+ | `from_gptq(qweight, qzeros, scales, g_idx=None, zero_offset=1)` | GPTQ tensors -> native; act-order returns an activation `perm` |
62
+ | `from_awq(qweight, qzeros, scales)` | AutoAWQ GEMM tensors -> native |
63
+ | `dequant_w4(packed, scales, zeros, group_size)` | reference f32 dequantization |
64
+ | `quantize_activation(x)` | per-token absmax INT8 |
65
+ | `w4a8_linear(x, packed, scales, zeros, group_size, perm)` | int4 weights x int8 activations |
66
+ | `w8a8_linear(x, w_int8, scale_wt)` | int8 x int8, per-row weight scale |
67
+ | `fp16_gemv(x, w_f16)` | f16-storage weights widened to f32 in registers |
68
+ | `QuantLinear` | `nn.Module` wrapper |
69
+
70
+ `zero_offset=1` follows the AutoGPTQ storage convention (the stored zero
71
+ plus one is subtracted at dequant); pass 0 for checkpoints without it.
72
+
73
+ ## Performance
74
+
75
+ Raspberry Pi 5 (4x Cortex-A76 2.4 GHz, SDOT path) and Raspberry Pi 4
76
+ Model B (4x Cortex-A72 1.8 GHz, NEON path), 64-bit Raspberry Pi OS,
77
+ torch 2.13 CPU, group_size 128:
78
+
79
+ | op, shape (M, N, K) | Pi 5 | Pi 4 |
80
+ |---|---|---|
81
+ | w4a8 1, 4096, 4096 | 1.16 ms | 4.08 ms |
82
+ | w4a8 1, 11008, 4096 | 2.98 ms | 11.7 ms |
83
+ | w4a8 1, 4096, 11008 | 3.57 ms | 10.4 ms |
84
+ | w4a8 128, 4096, 4096 | 23.2 ms | 209 ms |
85
+ | w8a8 1, 4096, 4096 | 1.46 ms | 5.68 ms |
86
+ | fp16 1, 4096, 4096 | 2.80 ms | 13.5 ms |
87
+
88
+ Decode (M=1) streams packed weights at 8-12 GB/s on the Pi 5, the board's
89
+ memory ceiling; M=128 runs 185 GOPS on the SDOT tier.
90
+
91
+ ## Requirements
92
+
93
+ - K divisible by group_size; group_size a multiple of 32 (64 and 128 tested).
94
+ - bf16 or f32 activations; output follows the input dtype.
95
+ - GPTQ act-order (`g_idx`) is supported through an activation permutation
96
+ applied per forward.
97
+ - x86-64 runs the scalar path: correct, not tuned.
98
+
99
+ ## Scope
100
+
101
+ Hub-loadable quantized linears for transformers-on-CPU, aimed at boards
102
+ where a dedicated llama.cpp runtime was previously the only usable path.
103
+ llama.cpp remains faster as a standalone runtime; this pack is for models
104
+ composed in Python.
105
+
106
+ ## License
107
+
108
+ Apache-2.0.