File size: 13,383 Bytes
9e7cac1 725119e 9e7cac1 725119e 9e7cac1 725119e 299d1df 725119e 299d1df 725119e 299d1df 725119e 299d1df ca6e3a9 299d1df 725119e 299d1df 725119e ca6e3a9 725119e 299d1df 725119e 299d1df ca6e3a9 299d1df 725119e 299d1df 725119e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 | ---
library_name: kernels
license: apache-2.0
tags:
- kernel
- webgpu
- wgsl
---
# ai.onnx.QLinearMatMul
`ai.onnx` · standard ONNX operator · ONNX opset ≥ 21
## Description
Quantized matrix multiplication of tensors `a` and `b` with supported ONNX `MatMul` vector and batch-broadcasting shapes. Each operand and the output carry their own scale and zero point; the result is rounded to nearest even and saturated to the output type. This package requires single-element rank-1 quantization tensors for `a` and `y`. For `b`, it supports single-element rank-1 tensors and selected per-column layouts; per-row quantization of `a` is unsupported.
See the [ONNX `QLinearMatMul` spec](https://onnx.ai/onnx/operators/onnx__QLinearMatMul.html) for the reference semantics.
## Inputs
| Name | Logical dtype | Rank | Shape | Description | Presence |
| --- | --- | --- | --- | --- | --- |
| `a` | `TA` | — | — | N-dimensional quantized matrix a. | required |
| `a_scale` | `TF` | `1` | — | Per-tensor scale of quantized input `a`, supplied as a single-element rank-1 tensor. | required |
| `a_zero_point` | `TA` | `1` | — | Zero point of quantized input `a`; must have the same shape as `a_scale`. | required |
| `b` | `TB` | — | — | N-dimensional quantized matrix b. | required |
| `b_scale` | `TF` | — | — | Scale of quantized input `b`: a single-element rank-1 tensor for per-tensor quantization, a length-N vector for a rank-2 matrix, or `[batch, 1, N]` for a rank-3 matrix with per-column quantization. | required |
| `b_zero_point` | `TB` | — | — | Zero point of quantized input `b`: a single-element rank-1 tensor, a length-N vector or `[1, N]` for a rank-2 matrix, or `[batch, 1, N]` for a rank-3 matrix. Rank-2 per-column scales and zero points may independently use their per-tensor forms. | required |
| `y_scale` | `TF` | `1` | — | Per-tensor scale of quantized output `y`, supplied as a single-element rank-1 tensor. | required |
| `y_zero_point` | `TY` | `1` | — | Per-tensor zero point of quantized output `y`, supplied as a single-element rank-1 tensor. | required |
## Outputs
| Name | Logical dtype | Rank | Shape | Description | Presence |
| --- | --- | --- | --- | --- | --- |
| `y` | `TY` | derived | ONNX MatMul result of `a` and `b` | Quantized matrix multiply result of a * b. | required |
## Type constraints
| Variable | Allowed dtypes |
| --- | --- |
| `TA` | `uint8`, `int8` |
| `TB` | `uint8`, `int8` |
| `TY` | `uint8`, `int8` |
| `TF` | `float32`, `float16` |
## Implementation variants
One implementation is selected per call from the device capabilities, the request shapes and the dtypes; these notes say what each one covers.
- `sgmat_precast_scalar_f16` — Prepare exact f16 representations of adjusted byte integers once, then load them directly into f32-accumulating subgroup matrices. Row groups follow matrix geometry, output rows, invocation limits and workgroup storage; bounded partial sums retain exact modular int32 arithmetic.
- `portable_precast_scalar_f16` — Prepare lossless f16 representations of adjusted byte integers, then use vector workgroup GEMM with exact bounded f32 partials and modular int32 totals. Geometry follows output rows, invocation and storage limits; f32 storage requires enough row reuse to amortize preparation.
- `sgmat_precast_scalar` — Prepare exact f32 representations of adjusted byte integers once, then load them directly into f32-accumulating subgroup matrices. Row groups follow matrix geometry, output rows, invocation limits and workgroup storage; bounded partial sums retain exact modular int32 arithmetic.
- `portable_precast_scalar` — Prepare lossless f32 representations of adjusted byte integers, then use vector workgroup GEMM with exact bounded f32 partials and modular int32 totals. Geometry follows output rows, invocation and storage limits; f32 storage requires enough row reuse to amortize preparation.
- `sgmat_precast_column_f16` — Prepare exact f16 representations of adjusted byte integers once, then load them directly into f32-accumulating subgroup matrices. Row groups follow matrix geometry, output rows, invocation limits and workgroup storage; bounded partial sums retain exact modular int32 arithmetic.
- `portable_precast_column_f16` — Prepare lossless f16 representations of adjusted byte integers, then use vector workgroup GEMM with exact bounded f32 partials and modular int32 totals. Geometry follows output rows, invocation and storage limits; f32 storage requires enough row reuse to amortize preparation.
- `sgmat_precast_column` — Prepare exact f32 representations of adjusted byte integers once, then load them directly into f32-accumulating subgroup matrices. Row groups follow matrix geometry, output rows, invocation limits and workgroup storage; bounded partial sums retain exact modular int32 arithmetic.
- `portable_precast_column` — Prepare lossless f32 representations of adjusted byte integers, then use vector workgroup GEMM with exact bounded f32 partials and modular int32 totals. Geometry follows output rows, invocation and storage limits; f32 storage requires enough row reuse to amortize preparation.
- `sgmat_precast_column_scale_f16` — Prepare exact f16 representations of adjusted byte integers once, then load them directly into f32-accumulating subgroup matrices. Row groups follow matrix geometry, output rows, invocation limits and workgroup storage; bounded partial sums retain exact modular int32 arithmetic.
- `portable_precast_column_scale_f16` — Prepare lossless f16 representations of adjusted byte integers, then use vector workgroup GEMM with exact bounded f32 partials and modular int32 totals. Geometry follows output rows, invocation and storage limits; f32 storage requires enough row reuse to amortize preparation.
- `sgmat_precast_column_scale` — Prepare exact f32 representations of adjusted byte integers once, then load them directly into f32-accumulating subgroup matrices. Row groups follow matrix geometry, output rows, invocation limits and workgroup storage; bounded partial sums retain exact modular int32 arithmetic.
- `portable_precast_column_scale` — Prepare lossless f32 representations of adjusted byte integers, then use vector workgroup GEMM with exact bounded f32 partials and modular int32 totals. Geometry follows output rows, invocation and storage limits; f32 storage requires enough row reuse to amortize preparation.
- `sgmat_precast_column_zero_f16` — Prepare exact f16 representations of adjusted byte integers once, then load them directly into f32-accumulating subgroup matrices. Row groups follow matrix geometry, output rows, invocation limits and workgroup storage; bounded partial sums retain exact modular int32 arithmetic.
- `portable_precast_column_zero_f16` — Prepare lossless f16 representations of adjusted byte integers, then use vector workgroup GEMM with exact bounded f32 partials and modular int32 totals. Geometry follows output rows, invocation and storage limits; f32 storage requires enough row reuse to amortize preparation.
- `sgmat_precast_column_zero` — Prepare exact f32 representations of adjusted byte integers once, then load them directly into f32-accumulating subgroup matrices. Row groups follow matrix geometry, output rows, invocation limits and workgroup storage; bounded partial sums retain exact modular int32 arithmetic.
- `portable_precast_column_zero` — Prepare lossless f32 representations of adjusted byte integers, then use vector workgroup GEMM with exact bounded f32 partials and modular int32 totals. Geometry follows output rows, invocation and storage limits; f32 storage requires enough row reuse to amortize preparation.
- `sgmat_exact_scalar` — Exact integer products through bounded f32 matrix partials, fused with scalar or per-column requantization. Tile height follows workgroup memory and invocation limits.
- `sgmat_exact_column` — Exact integer products through bounded f32 matrix partials, fused with scalar or per-column requantization. Tile height follows workgroup memory and invocation limits.
- `sgmat_exact_column_scale` — Exact integer products through bounded f32 matrix partials, fused with scalar or per-column requantization. Tile height follows workgroup memory and invocation limits.
- `sgmat_exact_column_zero` — Exact integer products through bounded f32 matrix partials, fused with scalar or per-column requantization. Tile height follows workgroup memory and invocation limits.
- `portable_accurate_dp4a_rank2_scalar_quantization` — Uses tiled integer accumulation and accurate requantization when a fused path is unavailable, including devices whose storage-binding limit cannot accommodate it. Native packed dot products are used when available.
- `portable_accurate_dp4a_rank2_per_column_quantization` — Uses tiled integer accumulation and accurate requantization when a fused path is unavailable, including devices whose storage-binding limit cannot accommodate it. Native packed dot products are used when available.
- `portable_accurate_dp4a_rank2_per_column_scale_scalar_zp` — Uses tiled integer accumulation and accurate requantization when a fused path is unavailable, including devices whose storage-binding limit cannot accommodate it. Native packed dot products are used when available.
- `portable_accurate_dp4a_rank2_scalar_quantization_tail` — Uses tiled integer accumulation and accurate requantization when a fused path is unavailable, including devices whose storage-binding limit cannot accommodate it. Native packed dot products are used when available.
- `portable_accurate_dp4a_rank2_per_column_scale_scalar_zp_tail` — Uses tiled integer accumulation and accurate requantization when a fused path is unavailable, including devices whose storage-binding limit cannot accommodate it. Native packed dot products are used when available.
- `portable_accurate_dp4a_rank3_scalar_quantization` — Uses tiled integer accumulation and accurate requantization when a fused path is unavailable, including devices whose storage-binding limit cannot accommodate it. Native packed dot products are used when available.
- `portable_accurate_dp4a_rank4_scalar_quantization` — Uses tiled integer accumulation and accurate requantization when a fused path is unavailable, including devices whose storage-binding limit cannot accommodate it. Native packed dot products are used when available.
- `portable_accurate_dp4a_rank2_scalar_scale_per_column_zp` — Uses tiled integer accumulation and accurate requantization when a fused path is unavailable, including devices whose storage-binding limit cannot accommodate it. Native packed dot products are used when available.
- `portable_accurate_dp4a_rank2_scalar_scale_per_column_zp_tail` — Uses tiled integer accumulation and accurate requantization when a fused path is unavailable, including devices whose storage-binding limit cannot accommodate it. Native packed dot products are used when available.
## Device requirements
Some implementation variants require `subgroup-matrix`, `shader-f16`, and `subgroups`. These are route-specific capabilities, not package-wide requirements; availability also depends on the request shape and dtype.
## Files
- [`metadata.json`](build/webgpu/metadata.json) — kernel metadata (id, digests, per-variant templates, provenance)
- [`manifest.json`](build/webgpu/manifest.json) — the op contract (source of truth)
- [`test.json`](build/webgpu/test.json) — correctness cases
- [`bench.json`](build/webgpu/bench.json) — benchmark cases
- [`qlinear-matmul-accumulate.wgsl.jinja`](build/webgpu/qlinear-matmul-accumulate.wgsl.jinja)
- [`qlinear-matmul-requantize.wgsl.jinja`](build/webgpu/qlinear-matmul-requantize.wgsl.jinja)
- [`quant-dp4a-matmul.wgsl.jinja`](build/webgpu/quant-dp4a-matmul.wgsl.jinja)
- [`quant-exact-matrix.wgsl.jinja`](build/webgpu/quant-exact-matrix.wgsl.jinja)
- [`quant-exact-portable.wgsl.jinja`](build/webgpu/quant-exact-portable.wgsl.jinja)
- [`quant-exact-prepare.wgsl.jinja`](build/webgpu/quant-exact-prepare.wgsl.jinja)
- [`quant-matmul-accumulate-rank4.wgsl.jinja`](build/webgpu/quant-matmul-accumulate-rank4.wgsl.jinja)
## Use with `@huggingface/kernels`
```sh
npm install --save-exact @huggingface/kernels@0.0.1-preview.3
```
Required output shapes and logical data types are inferred from the supplied inputs and attributes; result tensors are allocated automatically.
The `version: 1` option selects the published kernel contract; it is independent of any operator opset, contrib `since_version`, or model version.
It follows the `v1` branch as fixes land. To pin exact artifact bytes, pass a 40-character commit `revision` instead of `version`.
Replace each `*Data` placeholder with a typed array containing the corresponding input data.
```js
import { getKernel } from "@huggingface/kernels";
const kernel = await getKernel("webgpu-kernels/ai.onnx.QLinearMatMul", { version: 1 });
const { y } = await kernel({
a: { data: aData, shape: [1, 1] },
a_scale: { data: a_scaleData, shape: [1] },
a_zero_point: { data: a_zero_pointData, shape: [1] },
b: { data: bData, shape: [1, 2] },
b_scale: { data: b_scaleData, shape: [1] },
b_zero_point: { data: b_zero_pointData, shape: [1] },
y_scale: { data: y_scaleData, shape: [1] },
y_zero_point: { data: y_zero_pointData, shape: [1] },
});
```
|